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Abstract 


The  complex  and  dynamic  nature  of  various  types  of  operations  pose  specific  cognitive  challenges 
on  the  decision-making  process  that  the  current  training  regiment  of  military  commanders  does  not 
directly  address.  Therefore,  DRDC  Toronto  is  interested  in  researching  training  techniques  to 
prepare  Canadian  Forces  (CF)  commanders  and  staff  for  decision-making  in  such  complex  and 
dynamic  environments  (12sk).  This  report  provides  a  review  of  relevant  DDM  literature  and  mental 
models  literature  as  it  relates  to  DDM. 

DDM  consists  of  (1)  decision  maker(s)  (2)  in  a  complex  environment  (3)  attempting  to  accomplish 
one  or  more  tasks.  DDM  is  required  in  environments  with  high  risk  and  complexity,  and  involves 
the  performance  of  tasks  requiring  multiple  steps,  that  are  inherently  time  sensitive,  interdependent, 
and  which  exert  influence  over  the  surrounding  environment  as  well  as  being  influenced  by  it. 
Dynamic  decision-making  has  been  explored  from  different  perspectives,  including  systems  theory, 
psychology,  and  control  theory  from  the  engineering  domain.  These  perspectives  put  varying 
amounts  of  focus  on  different  aspects  of  DDM.  What  is  common  to  all  of  these  approaches  are  the 
assumptions  that  whether  forming  models  of  complex  systems  or  making  intuitive  decisions  based 
on  very  little  information,  people  tend  to  form  some  sort  of  mental  model  to  undertake  DDM. 

At  a  broad  level,  mental  models  can  be  described  as  personal  mental  representations  of  our  world. 
Although  there  is  no  one  agreed  definition  of  mental  models,  they  are  generally  recognized  to  serve 
three  key  functions:  to  describe,  to  predict,  and  to  explain  our  world.  The  aim  of  this  report  was  to 
explore  how  mental  models  are  understood  across  the  propositional  logic,  physical  systems, 
situation  model,  and  system  dynamics  perspectives.  Few  similarities  in  the  descriptions  of  mental 
models  were  found  between  the  domains  reviewed  in  this  report.  In  addition,  mental  models  were 
found  to  be  subject  to  a  range  of  shortcomings  that  impact  the  accuracy  of  mental  models  (e.g., 
limited  cognitive  capacity,  erroneous  causal  links,  based  on  incomplete  information,  systemic 
errors). 

The  extent  to  which  DDM  could  be  supported  by  each  of  the  unique  approaches  to  understanding 
mental  models  was  assessed,  as  well  as  an  exploration  of  how  well  the  use  of  mental  models  would 
be  likely  to  support  the  dynamic  decision-making  process.  This  review  closes  with  an  outline  of  the 
mental  models  research  that  needs  to  be  better  explored  to  serve  DDM,  as  well  as  an  outline  of  the 
general  mental  models  research  questions  that  should  be  addressed  in  future  research. 
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Resume 


Le  caractere  complexe  et  dynamique  de  certains  types  d’ operations  pose  des  problemes  cognitifs 
particuliers  an  processus  de  prise  de  decisions  que  I’instruction  actuelle  des  commandants 
militaires  ne  traite  pas  directement.  RDDC  Toronto  est  done  a  la  recherche  de  techniques 
d’instruction  qui  permettront  de  former  les  commandants  et  les  membres  d’etat-major  des  Forces 
canadiennes  (FC)  a  la  prise  de  decisions  dans  des  environnements  complexes  et  dynamiques 
(12sk).  Le  present  compte  rendu  renferme  un  aperfu  de  la  documentation  sur  la  FDD  et  sur  les 
modeles  mentaux  associes  a  la  FDD. 

La  FDD  implique  la  participation  (1)  d’un  decideur,  (2)  dans  un  environnement  complexe,  (3)  qui 
tente  d’executer  une  ou  plusieurs  taches.  La  FDD  est  necessaire  dans  des  environnements 
complexes  ou  le  risque  est  eleve.  Elle  implique  I’execution  de  taches  requerant  des  etapes 
multiples,  pour  lesquelles  le  temps  est  important,  qui  sont  interdependantes  et  qui  exercent  une 
influence  sur  le  milieu  environnant,  tout  en  etant  influencees  par  lui.  La  prise  de  decisions 
dynamique  a  ete  exploree  a  partir  de  differents  points  de  vue,  notamment  la  theorie  des  systemes,  la 
psychologic  et  la  t  heorie  de  controle  dans  le  domaine  de  I’ingenierie.  Ces  points  de  vue  mettent 
F accent,  a  divers  degres,  sur  differents  aspects  de  la  FDD.  Une  hypothese  est  commune  a  toutes  ces 
approches  :  qu’il  s’agisse  de  former  des  modeles  de  systemes  complexes  ou  de  prendre  des 
decisions  intuitives  a  partir  de  tres  peu  de  renseignements,  les  gens  ont  tendance  a  developper  une 
certaine  forme  de  modele  mental  pour  effectuer  une  FDD. 

De  fa9on  generale,  on  pent  decrire  les  modeles  mentaux  comme  des  representations  mentales  de 
notre  monde.  Bien  qu’il  n’y  ait  aucun  consensus  sur  la  definition  des  modeles  mentaux,  ces 
demiers  sont  generalement  reconnus  pour  etre  associes  a  trois  fonctions  cles  :  decrire,  predire  et 
expliquer  le  monde  qui  nous  entoure.  Le  present  compte  rendu  a  pour  but  d’ explorer  comment  les 
modeles  mentaux  sont  compris  du  point  de  vue  de  la  logique  des  propositions,  des  systemes 
causals,  du  modele  de  situation  et  de  la  dynamique  des  systemes.  On  a  trouve  peu  de  similitudes 
dans  les  descriptions  des  modeles  mentaux  entre  les  domaines  examines  dans  la  presente  analyse. 
En  outre,  on  a  decouvert  que  des  modeles  mentaux  etaient  sujets  a  une  variete  de  lacunes 
infiuenfant  leur  exactitude  (p.  ex.,  capacite  cognitive  limitee,  liens  causals  errones  fondes  sur  des 
renseignements  incomplets,  erreurs  systemiques). 

On  a  evalue  jusqu’a  quel  point  la  FDD  pouvait  etre  appuyee  par  chacune  des  approches  uniques  a 
la  comprehension  des  modeles  mentaux,  de  meme  qu’on  a  explore  jusqu’a  quel  point  I’utilisation 
de  modeles  mentaux  appuie  le  processus  de  prise  de  decisions  dynamique.  E’examen  se  termine  sur 
les  aspects  de  la  recherche  sur  les  modeles  mentaux  devant  etre  mieux  explores  pour  servir  la  FDD, 
de  meme  que  sur  un  aperfu  des  questions  generates  relatives  a  la  recherche  sur  les  modeles 
mentaux  qui  devraient  etre  traitees. 
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Executive  Summary 


The  complex  and  dynamic  nature  of  operations-other-than-war  (OOW)  (e.g.,  peace  support,  the  3- 
block  war  concept)  in  which  Canada  and  allied  nations  are  increasingly  involved  requires  Canadian 
Forces  (CF)  officers  to  call  upon  high-level  dynamic  decision-making  (DDM)  skills  to  an 
unprecedented  degree,  especially  at  the  strategic  and  operational  levels  (Rehak,  Lamoureux,  & 

Bos,  2006).  The  complex  and  dynamic  nature  of  these  various  types  of  operations  pose  specific 
cognitive  challenges  on  the  decision-making  process  that  the  current  training  regiment  of  military 
commanders  does  not  directly  address.  Therefore,  DRDC  Toronto  is  interested  in  researching 
training  techniques  to  prepare  Canadian  Forces  (CF)  commanders  and  staff  for  decision-making  in 
such  complex  and  dynamic  environments  (12sk).  This  report  provides  a  review  of  relevant  DDM 
literature  and  mental  models  literature  as  it  relates  to  DDM. 

DDM  consists  of  (1)  decision  maker(s)  (2)  in  a  complex  environment  (3)  attempting  to  accomplish 
one  or  more  tasks.  DDM  is  required  in  environments  with  high  risk  and  complexity,  and  involves 
the  performance  of  tasks  requiring  multiple  steps,  that  are  inherently  time  sensitive,  interdependent, 
and  which  exert  influence  over  the  surrounding  environment  as  well  as  being  influenced  by  it. 
Dynamic  decision-making  has  been  explored  from  different  perspectives,  including  systems  theory, 
psychology,  and  control  theory  from  the  engineering  domain.  These  perspectives  put  varying 
amounts  of  focus  on  different  aspects  of  DDM.  From  the  systems  theory  perspective,  for  example, 
the  focus  is  on  creating  analogies  that  attempt  to  simulate  the  processes  that  people  use  to  manage 
complex  systems.  These  analogies,  moreover,  represent  one  way  to  help  people  to  form  more 
accurate  models  of  the  system.  From  the  psychological  perspective,  on  the  other  hand,  the  focus  is 
on  the  sets  of  choices  that  people  make  as  they  attempt  to  make  complex  decisions.  Control 
theory’s  primary  emphasis  is  on  the  role  of  feedback  while  managing  a  complex  system.  What  is 
common  to  all  of  these  approaches  are  the  assumptions  that  whether  forming  models  of  complex 
systems  or  making  intuitive  decisions  based  on  very  little  information,  people  tend  to  form  some 
sort  of  mental  model  to  undertake  DDM. 

At  a  broad  level,  mental  models  can  be  described  as  personal  mental  representations  of  our  world. 
Although  there  is  no  one  agreed  definition  of  mental  models,  they  are  generally  recognized  to  serve 
three  key  functions:  to  describe,  to  predict,  and  to  explain  our  world.  The  aim  of  this  report  was  to 
explore  how  mental  models  are  understood  across  a  range  of  relevant  perspectives.  The 
propositional  logic  perspective  argues  that  reasoning  depends  on  imagining  the  possibilities 
compatible  with  the  premises,  and  drawing  conclusions  from  these  mental  representations. 
Flowever,  there  are  a  number  of  factors  limiting  the  application  of  propositional  models  to  dynamic 
decision-making  (e.g.,  focus  on  a  very  small  subset  of  reasoning  tasks  that  involve  static  finite 
constructs,  no  interaction  between  effects  of  decision  makers  on  the  task  or  on  the  environment). 
The  physical  systems  perspective  defines  mental  models  as  the  models  people  use  in  reasoning 
about  the  physical  world.  Flowever,  the  literature  on  physical  systems  raises  a  number  of  issues 
pertaining  to  the  ability  of  mental  models  to  support  DDM  (e.g.,  it  is  unknown  how  such  mental 
models  are  created  or  work  in  DDM  environments).  The  situation  model  perspective  defines  a 
situation  model  a  dynamic  representation  of  a  person’s  knowledge  and  understanding  of  the  present 
state  of  a  system,  whereas  the  system  dynamics  perspective  describes  mental  models  as  internal 
conceptual  representation  of  an  external  system.  In  this  review,  few  similarities  in  the  descriptions 
of  mental  models  were  found  between  the  domains  reviewed  in  this  report.  This  finding  highlights 
the  challenges  to  understanding  mental  models  already  described  in  mental  models  literature  (i.e.. 
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lack  of  a  clear  definition,  difficulty  measuring  mental  models).  In  addition,  mental  models  were 
also  found  to  be  subject  to  a  range  of  shortcomings  that  impact  the  accuracy  of  mental  models  (e.g., 
limited  cognitive  capacity,  erroneous  causal  links,  based  on  incomplete  information,  systemic 
errors). 

The  extent  to  which  DDM  could  be  supported  by  each  of  the  unique  approaches  to  understanding 
mental  models  was  assessed.  With  respect  to  propositional  models,  it  was  difficult  to  see  any 
substantive  symmetries  between  this  perspective  of  mental  models  and  DDM.  Similarly,  little  of 
the  literature  discussion  mental  models  of  physical  systems  related  well  to  DDM.  As  well, 
inconsistencies  among  the  definition  of  mental  models  in  system  dynamics  makes  it  difficult  to 
clearly  understand  the  role  of  mental  models  in  dynamic  decision  making  The  situation  model 
perspective  was  assessed  to  be  conductive  to  supporting  DDM  as  such  models  allow  the  decision 
maker  to  incorporate  changes  to  the  system  into  their  situation  model  to  provide  them  with  an 
accurate  and  up-to-date  vision  of  system  status  and  function.  However,  research  suggests  that 
people  are  not  particularly  effective  at  creating  accurate  situation  models. 

This  report  also  explored  how  well  the  use  of  mental  models  would  be  likely  to  support  the 
dynamic  decision-making  process.  With  respect  to  the  dynamic  decision  maker,  the  mental  models 
literature  contains  vague  descriptions  of  the  exact  mental  models  processes  used  by  decision 
makers.  In  addition,  the  literature  suggests  a  number  of  inherent  limitations  of  decision-makers  in 
being  able  to  form  and  apply  mental  models,  such  as  cognitive  limitations,  mental  effort,  heuristics, 
and  biases.  Similarly,  the  nature  of  DDM  environments  (i.e.,  complexity,  feedback)  and  DDM 
tasks  (i.e.,  time,  uncertainty)  are  unlikely  to  support  the  use  of  mental  models  to  make  decisions. 
However,  this  does  not  mean  that  mental  models  cannot  be  helpful,  simply  that  the  limitations  of 
whatever  models  can  be  formed  should  be  recognized. 

This  review  closes  with  two  tables  that  summarize  gaps  in  the  literature.  The  first  table  outline  the 
mental  models  research  that  needs  to  be  better  explored  to  serve  DDM,  and  the  second  table  outline 
general  mental  models  research  questions  that  should  be  addressed. 
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Le  caractere  complexe  et  dynamique  des  operations  autres  que  la  guerre  (OAG)  (soutien  de  la  paix, 
concept  de  la  guerre  a  trois  volets,  etc.)  auxquelles  le  Canada  et  les  pays  allies  participent  de  plus 
en  plus,  exige  que  les  officiers  des  Forces  canadiennes  (FC)  fasses  appel  a  des  competences 
superieures  en  matiere  de  prise  de  decisions  dynamique  a  un  niveau  sans  precedent,  en  particulier 
aux  plans  operationnel  et  strategique  (Rehak,  Lamoureux,  &  Bos,  2006).  Le  caractere  complexe  et 
dynamique  de  ces  divers  types  d’ operations  pose  des  problemes  cognitifs  particuliers  au  processus 
de  prise  de  decisions  que  I’instruction  actuelle  des  commandants  militaires  ne  traite  pas 
directement.  RDDC  Toronto  est  done  a  la  recherche  de  techniques  d’instruction  qui  permettront  de 
former  les  commandants  et  les  membres  d’etat-major  des  Forces  canadiennes  a  la  prise  de 
decisions  dans  des  environnements  complexes  et  dynamiques  (12sk).  Le  present  compte  rendu 
renferme  un  aperfu  de  la  documentation  sur  la  FDD  et  sur  les  modeles  mentaux  associes  a  la  FDD. 

La  FDD  implique  la  participation  (1)  d’un  decideur,  (2)  dans  un  environnement  complexe,  (3)  qui 
tente  d’executer  une  ou  plusieurs  taches.  La  FDD  est  necessaire  dans  des  environnements 
complexes  ou  le  risque  est  eleve.  Elle  implique  I’execution  de  taches  requerant  des  etapes 
multiples,  pour  lesquelles  le  temps  est  important,  qui  sont  interdependantes  et  qui  exercent  une 
influence  sur  le  milieu  environnant,  tout  en  etant  influencees  par  lui.  La  prise  de  decisions 
dynamique  a  etc  exploree  a  partir  de  differents  points  de  vue,  notamment  la  theorie  des  systemes,  la 
psychologic  et  la  theorie  de  controle  dans  le  domaine  de  I’ingenierie.  Ces  perspectives  mettent 
F accent,  a  divers  degres,  sur  differents  aspects  de  la  FDD.  Du  point  de  vue  de  la  theorie  des 
systemes  par  exemple,  I’accent  porte  sur  la  creation  d’analogies  qui  tentent  de  simuler  les 
processus  que  les  gens  utilisent  pour  gerer  des  systemes  complexes.  De  plus,  ces  analogies 
representent  une  fafon  d’ aider  les  gens  a  former  des  modeles  plus  precis  du  systeme.  Far  ailleurs, 
du  point  de  vue  psychologique,  I’accent  porte  sur  les  ensembles  de  choix  que  les  gens  peuvent  faire 
lorsqu’ils  essaient  de  prendre  des  decisions  complexes.  La  theorie  du  controle  met  principalement 
I’accent  sur  le  role  de  la  retroaction  dans  la  gestion  d’un  systeme  complexe.  Une  hypothese  est 
commune  a  toutes  ces  approches  :  qu’il  s’agisse  de  former  des  modeles  de  systemes  complexes  ou 
de  prendre  des  decisions  intuitives  a  partir  de  tres  peu  de  renseignements,  les  gens  ont  tendance  a 
developper  une  certaine  forme  de  modele  mental  pour  effectuer  une  FDD. 

De  fafon  generale,  on  pent  decrire  les  modeles  mentaux  comme  des  representations  mentales  de 
notre  monde.  Bien  qu’il  n’y  ait  aucun  consensus  sur  la  definition  des  modeles  mentaux,  ces 
demiers  sont  generalement  reconnus  pour  etre  associes  a  trois  fonctions  cles  :  decrire,  predire  et 
expliquer  le  monde  qui  nous  entoure.  Le  present  rapport  avait  pour  but  de  decouvrir  de  quelle  fafon 
les  modeles  mentaux  sont  compris  parmi  toute  une  gamme  de  points  de  vue  pertinents.  Du  point  de 
vue  de  la  logique  des  propositions,  on  allegue  que  le  raisonnement  depend  de  1’ imagination  des 
possibilites  compatibles  avec  les  premisses  et  des  conclusions  que  Ton  tire  de  ces  representations 
mentales.  II  y  a  cependant  un  certain  nombre  de  facteurs  qui  limitent  I’application  de  modeles 
propositionnels  a  la  prise  de  decisions  dynamique  (p.  ex.,  attention  portee  a  un  tres  petit 
sous-ensemble  de  taches  de  raisonnement  impliquant  des  concepts  statiques  finis,  absence 
d’interaction  entre  les  repercussions  des  decideurs  sur  la  tache  ou  1’ environnement).  La  perspective 
des  systemes  physiques  definit  les  modeles  mentaux  comme  etant  les  modeles  que  les  gens  utilisent 
dans  leur  raisonnement  sur  le  monde  physique.  Toutefois,  la  documentation  sur  les  systemes 
physiques  souleve  un  certain  nombre  de  questions  relatives  a  la  capacite  des  modeles  mentaux 
d’appuyer  la  FDD  (p.  ex.,  nous  ne  savons  pas  de  quelle  fafon  ces  modeles  mentaux  sont  crees  ou 


^{umwsystems®  Incorporated 


Mental  Models  and  DDM 


Page  v 


HUMANSrSTEMS 


fonctionnent  dans  le  contexte  de  la  PDD).  La  perspective  du  modele  de  situation  definit  un  modele 
de  situation  comme  etant  la  representation  dynamique  de  la  connaissance  et  de  la  comprehension 
qu’a  une  personne  de  I’etat  actuel  d’un  systeme,  alors  que  la  perspective  de  la  dynamique  d’un 
systeme  decrit  les  modeles  mentaux  comme  etant  la  representation  conceptuelle  interne  d’un 
systeme  externe.  Dans  la  presente  analyse,  on  a  trouve  peu  de  similitudes  dans  les  descriptions  des 
modeles  mentaux  entre  les  domaines  examines.  La  conclusion  met  en  lumiere  les  difficultes  a 
comprendre  les  modeles  mentaux  deja  decrits  dans  la  documentation  sur  les  modeles  mentaux  (p. 
ex.,  le  manque  d’une  definition  claire,  la  difficulte  a  mesurer  les  modeles  mentaux).  En  outre,  on  a 
decouvert  que  des  modeles  mentaux  etaient  sujets  a  une  variete  de  lacunes  ayant  une  influence  sur 
leur  exactitude  (p.  ex.,  capacite  cognitive  limitee,  liens  causals  errones  fondes  sur  des 
renseignements  incomplets,  erreurs  systemiques). 

On  a  evalue  jusqu’a  quel  point  la  PDD  pouvait  etre  appuyee  par  chacune  des  approches  uniques  a 
la  comprehension  des  modeles  mentaux.  En  ce  qui  a  trait  aux  modeles  operationnels,  il  a  ete 
difficile  de  trouver  des  symetries  substantielles  entre  cette  perspective  de  modeles  mentaux  et  la 
PDD.  De  la  meme  fafon,  une  partie  limitee  de  la  documentation  portant  sur  les  modeles  mentaux 
des  systemes  physiques  pouvait  etre  lice  adequatement  a  la  PDD.  De  plus,  certaines  incoherences 
dans  la  definition  des  modeles  mentaux  de  la  dynamique  des  systemes  rendent  difficile  la 
comprehension  claire  du  role  des  modeles  mentaux  dans  la  PDD.  On  a  estime  que  la  perspective  du 
modele  de  situation  etait  favorable  pour  appuyer  la  PDD  etant  donne  que  ces  modeles  permettent  a 
un  decideur  d’integrer  les  changements  d’un  systeme  a  son  modele  de  situation  de  maniere  a  lui 
foumir  une  vision  precise  et  a  jour  de  I’etat  et  de  la  function  du  systeme.  Toutefois,  la  recherche 
laisse  entendre  que  les  gens  ne  sont  pas  particulierement  efficaces  pour  creer  des  modeles  de 
situation  justes. 

Ee  compte  rendu  a  egalement  explore  jusqu’a  quel  point  I’utilisation  des  modeles  mentaux  appuie 
le  processus  de  PDD.  En  ce  qui  a  trait  aux  decideurs  dynamiques,  la  documentation  sur  les  modeles 
mentaux  ne  renferme  que  de  vagues  descriptions  des  processus  exacts  qu’ils  utilisent.  En  outre,  la 
documentation  signale  un  certain  nombre  de  restrictions  empechant  les  decideurs  de  former  et 
d’appliquer  des  modeles  mentaux,  notamment  des  restrictions  cognitives,  I’effort  mental,  les 
heuristiques  et  les  biais.  De  la  meme  maniere,  il  est  peu  probable  que  le  contexte  de  la  PDD 
(complexite,  retroaction)  et  les  taches  de  la  PDD  appuient  I’utilisation  de  modeles  mentaux  pour 
prendre  des  decisions.  Cela  ne  signifie  cependant  pas  que  les  modeles  mentaux  ne  peuvent  pas  etre 
utiles,  mais  simplement  que  les  limites  des  modeles  formes,  quels  qu’ils  soient,  devraient  etre 
reconnues. 

E’examen  se  termine  avec  deux  tableaux  qui  resument  les  lacunes  de  la  documentation.  Ee  premier 
tableau  passe  en  revue  la  recherche  des  modeles  mentaux  a  ameliorer  pour  servir  la  PDD  et  le 
deuxieme  tableau  met  en  relief  les  questions  generales  relatives  a  la  recherche  sur  les  modeles 
mentaux  qui  devraient  etre  traitees. 
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1.  Introduction 


1.1  Background 

The  complex  and  dynamic  nature  of  military  operations  in  which  Canada  and  allied  nations  are 
increasingly  involved  requires  Canadian  Forces  (CF)  officers  to  call  upon  high-level  dynamic 
decision-making  (DDM)  skills  to  an  unprecedented  degree,  especially  at  the  strategic  and 
operational  levels  (Rehak,  Lamoureux,  &  Bos,  2006).  The  complex  and  dynamic  nature  of  these 
various  types  of  operations  pose  specific  cognitive  challenges  on  the  decision-making  process  that 
the  current  training  regimen  of  military  commanders  does  not  directly  address.  Therefore,  DRDC 
Toronto  is  interested  in  researching  training  techniques  to  prepare  Canadian  Forces  (CF) 
commanders  and  staff  for  decision-making  in  such  complex  and  dynamic  environments. 

Dynamic  decision-making  (DDM)  generally  refers  to  situations  that  require  a  series  of  interrelated 
decisions  made  in  real  time  with  the  aim  of  controlling  or  influencing  a  situation.  This  is  in 
contrast  with  the  more  “static”  decision-making  approach  traditionally  researched  in  psychology. 

In  DDM,  the  state  of  the  problem  changes  continuously,  both  autonomously  and  as  a  consequence 
of  the  decision-maker’s  actions,  and  the  decision-making  environment  is  opaque  (i.e.,  it  is  not 
possible  for  a  decision  maker  to  know  all  aspects  or  variables  of  the  environment,  and  therefore 
some  characteristics  of  the  system  must  be  inferred)  (Brehmer,  1992). 

Much  of  the  research  conducted  on  DDM  assumes  that  mental  models  and  mental  simulation  skills, 
or  at  least  recognition  processes,  are  crucial  to  successful  DDM  (see,  e.g.,  Brehmer,  1990; 
Gonzalez,  Lerch,  &  Lebiere,  2003).  Flowever,  little  scientific  publications  on  DDM  make  reference 
to  any  psychological  theories  of  mental  models.  Furthermore  the  DDM  literature  does  not  provide 
much  information  on  the  cognitive  mechanisms  that  would  underlie  DDM.  This  dearth  is 
problematic,  especially  as  accounts  of  mental  models  vary  in  the  extent  to  which  mental  models 
rely  on  visual  representation  -  on  a  continuum  from  purely  visual  images  to  purely  propositional 
logic.  Although  it  is  difficult  to  see  how  purely  proposition-based  mental  models  might  be  suited  to 
DDM,  it  raises  the  question  of  what  form  do  mental  models  take  in  DDM,  and  what  role  do  they 
play  in  the  DDM  process? 

In  contrast  to  psychology,  the  cognitive  engineering  approach  to  supporting  DDM  has  relied  on  the 
system  dynamics  approach  to  modeling  complex  systems.  This  approach  assumes  that  presenting 
models  of  the  dynamics  of  typical  systems  (i.e.,  the  evolution  over  time  of  the  set  of  variables  that 
describe  it)  to  decision  makers  will  help  them  improve  their  DDM  skills.  These  models  might  be 
interactive  simulations  (microworlds)  or  simply  diagrams  of  the  system.  The  underlying  hypothesis 
of  this  approach  is  that  presenting  the  decision  makers  with  such  models  will  improve  their  mental 
model  of  the  system  and  its  dynamics.  Again,  despite  such  a  reliance  on  the  concept  of  mental 
model,  very  little  reference  is  made  to  the  literature  on  mental  models.  In  fact  some  research  in  the 
system  dynamics  field  has  identified  a  lack  of  consensus  among  systems  dynamics  practitioners  as 
to  what  mental  models  are  (Doyle  &  Ford,  1998). 

The  objective  of  this  work  is  to  understand  what  kinds  of  mental  models  or  representations  are 
involved  in  DDM,  and  the  role  they  play  in  the  DDM  process. 
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W771 1-0781  lO/OOl/TOR,  Call-up  No.  8110-22.  The  Scientific  Authority  (SA)  for  this  project  is 
Dr.  Marie-Eve  Jobidon. 

1 .2  Scope  and  objectives 

The  overall  aim  of  this  work/project  is  to  review  literature  on  mental  models  and  DDM  in  order  to 
better  understand  the  form  that  mental  models  take  in  DDM  and  the  role  that  they  play  in  the  DDM 
process.  More  specifically,  the  objectives  are: 

1 .  To  review  relevant  DDM  literature. 

2.  To  review  the  literature  on  mental  models  as  it  relates  to  DDM. 

3.  To  determine  gaps,  if  any,  in  the  literature  on  mental  models  regarding  DDM. 

4.  To  construct  (or  build  upon  an  existing)  conceptual  framework  for  relating  mental  models 
to  DDM,  guiding  areas  for  future  research  on  mental  models  relative  to  DDM,  and 
identifying  constraints  and  implications  for  DDM  training. 

1.3  Outline  of  report 

This  report  provides  a  review  of  relevant  DDM  literature  and  mental  models  literature  as  it  relates 
to  DDM. 

This  report  has  six  main  sections: 

1 .  Introduction; 

2.  Method; 

3.  Dynamic  Decision-Making; 

4.  Mental  Models; 

5.  Mental  Models  and  Dynamic  Decision-making;  and 

6.  Future  Research  Recommendations  based  on  Gaps  in  the  Literature. 

These  sections  encapsulate  the  work  items  described  in  the  Statement  of  Work  (SOW). 
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2.1  Literature  Reviews 

A  structured  process  was  applied  in  conducting  the  literature  review.  It  included  finding  the 
relevant  articles  (either  through  their  provision  by  the  SA  or  through  extensive  searching),  filtering 
relevant  articles,  reviewing  the  articles  and  then  performing  the  final  analysis  and  synchronization 
of  the  status  of  the  literature.  These  steps  were  conducted  twice,  once  focusing  on  dynamic 
decision  making  and  once  focusing  on  mental  models.  More  details  on  finding  and  filtering  the 
articles  can  be  found  below. 

2.1.1  Dynamic  Decision-making 

The  first  part  of  the  literature  review  looked  to  understand  and  document  the  characteristics  and 
research  status  of  DDM.  After  discussions  with  the  SA  at  the  start-up  meeting,  it  was  agreed  that 
the  SA  would  provide  HSI®  with  the  appropriate  articles  to  be  reviewed  for  DDM.  In  total,  the  SA 
provided  HSI®  with  1 9  articles  on  DDM  to  be  reviewed.  This  list  was  augmented  with  an 
additional  three  articles  that  were  found  while  conducting  the  searches  detailed  below. 

2.1.2  Mental  Models  in  Dynamic  Decision-making 

Building  on  the  previous  section,  the  second  phase  of  the  project  looked  at  mental  models.  This 
included  literature  on  mental  models  in  general  as  well  as  mental  models  applied  in  DDM.  They 
keywords  used  in  the  search  can  be  found  below  in  Table  1. 
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Table  1:  Keywords 


Core  Concept 

Primary  Keywords 

Secondary  Keywords 

Decision-making 

Reasoning,  judgement,  probiem 
soiving,  intuitive,  rationai,  naturaiistic. 

Creativity,  deductive,  inductive,  inference, 
anaiytic,  counter-factuai  reasoning 

Dynamic  Decision-making 

risk,  compiexity,  uncertainty, 
ambiguity,  probiem  space 

Systems  theory 

System  dynamics,  compiex  systems, 
D3M  (distributed  dynamic  decision¬ 
making) 

Noniinearity,  stock  and  flow,  feedback  ioops, 
causai  ioop,  emergence,  emergent  behaviour 

Contexts 

Miiitary,  crisis,  operationai,  strategic, 
command  and  controi,  C2,  effects 
based  operations,  order  effects 

Uncertain,  criticai,  pressure,  risk 

Mentai  Modeis 

Cognitive  mechanisms,  probabiiistic 
information  processing,  cognitive 
modei,  representations,  mentai 
processes,  iogic 

Psychoiogicai  theories,  stock  and  flow, 
framework,  iearning  theory,  mentai 
representation,  mentai  simuiation 

Physicai  systems 

Situation  modeis 

isomorphism 

Cognitive 

mentai,  knowiedge,  inteiiectuai,  abiiity 

sensory,  perception,  attention, 
workioad 

iearning 

training 

memory 

forgetting,  retention,  recaii,  maintenance 

ianguage,  communication 

In  conducting  the  search  for  mental  models  articles,  the  primary  keywords  were  searched 
independently  and  were  paired  with  “mental  models”. 

2.1. 2.1  Databases 

A  variety  of  databases  were  searched.  These  are  summarized  below  in  Table  2. 
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Table  2:  Databases  searched 


Database 

Description 

PsyclNFO 

The  PsyclNFO  database  Is  a  collection  of  electronically  stored  bibliographic  references,  often  with 
abstracts  or  summaries,  to  psychological  literature  from  the  1800s  to  the  present.  The  available  literature 
Includes  material  published  In  50  countries,  but  Is  all  presented  In  English.  Books  and  chapters  published 
worldwide  are  also  covered  In  the  database,  as  well  as  technical  reports  and  dissertations  from  the  last 
several  decades. 

NTIS 

NTIS  Is  an  agency  for  the  U.S.  Department  of  Commerce’s  Technology  Administration.  It  Is  the  official 
source  for  government  sponsored  U.S.  and  worldwide  scientific,  technical,  engineering  and  business 
related  Information.  The  400,000  article  database  can  be  searched  for  free  at  the  www.ntls.qov.  Articles 
can  be  purchased  from  NTIS  at  costs  depending  on  the  length  of  the  article. 

CISTI 

CISTI  stands  for  the  Canada  Institute  for  Scientific  and  Technical  Information.  It  Is  the  library  for  the 

National  Research  Council  of  Canada  and  a  world  source  for  Information  In  science,  technology, 
engineering  and  medicine.  The  database  Is  searchable  on-line  at  cat.cistl.nrc.ca.  Articles  can  be 
ordered  from  CISTI  for  a  fee  of  approximately  $12. 

STINET 

STINET  Is  a  publlcly-avallable  database  (stlnet.dtlc.mil)  which  provides  access  to  citations  of  documents 
such  as:  unclassified  unlimited  documents  that  have  been  entered  Into  the  Defence  Technology  Technical 
Reports  Collection  (e.g.,  dissertations  from  the  Naval  Postgraduate  School),  the  Air  University  Library 

Index  to  Military  Periodicals,  Staff  College  Automated  Military  Periodical  Index,  Department  of  Defense 

Index  to  Specifications  and  Standards,  and  Research  and  Development  Descriptive  Summaries.  The  full- 
text  electronic  versions  of  many  of  these  articles  are  also  available  from  this  database. 

Google 

Scholar 

The  World  Wide  Web  was  searched  using  the  Google  Scholar  search  engines  (scholar.google.com). 

DRDC 

Research 

Reports 

DRDC  Defence  Research  Reports  Is  a  database  of  scientific  and  technical  research  produced  over  the 
past  6-  years  by  and  for  the  Defence  Research  &  Development  Canada.  It  Is  available  online  at 
pubs.drdc-rddc.gc.ca/pubdocs/pcow1_e.html. 

2.1. 2.2  Search  stra tegy 


To  maintain  a  record  of  the  process,  the  following  information  was  documented  in  a  spreadsheet 
throughout  the  search  process: 

•  Database  searched  (e.g.,  Psych  Info); 

•  Keyword  combination  (e.g.,  Non  intrusive  AND  attenti*); 

•  Number  of  hits; 

•  Number  of  possible  articles; 

•  Articles  downloaded;  and 

•  Articles/books  that  require  purchase. 

2. 1.2.3  Selection  and  review  of  articles 

HSI  identified  41  possible  articles  for  the  mental  models  review.  These  articles  were  then  narrowed 
down  to  1 8  articles,  which  were  presented  to  the  SA.  Through  further  iterations  of  searching  and 
filtering,  five  of  the  original  1 8  articles  presented  to  the  SA  were  removed  from  the  list  as  an 
additional  13  articles  were  found  to  be  more  relevant.  We  were  also  asked  to  use  one  specific 
article  from  Annex  A  to  the  Statement  of  Work.  In  total,  34  articles  were  reviewed  for  the  mental 
models  literature  section. 
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3.  Dynamic  Decision-making 


3.1  Defining  Dynamic  Decision-making 

There  are  four  key  elements  that  define  dynamic  decision-making  (DDM)  (Brehmer  &  Allard, 

1991;  Busemeyer,  1999;  Clancy,  Elliott,  Ley,  Omodei,  Wearing,  McLennen  &  Thorsteinsson, 

2003;  Fu  &  Gonzalez  2006): 

•  There  are  a  series  of  decisions; 

•  These  decisions  are  interdependent; 

•  The  environment  changes  both  autonomously  and  as  a  function  of  the  decision  maker’s 
actions;  and, 

•  Timing  is  a  key  element,  where  decision  makers  have  little  control  over  exactly  when 
dynamic  decisions  must  be  made  (Brehmer,  2000). 

These  decisions  are  not  made  autonomously,  they  must  be  made  by  decision  makers.  Further,  the 
decision  maker  is  acting  out  of  desire  to  accomplish  something  -  to  reach  a  goal  or  complete  a  task. 
This  task  is  being  attempted  within  the  confines  of  its  environment,  upon  which  the  decision  maker 
has  little  or  no  control.  Thusly,  breaking  DDM  into  its  constituent  parts,  DDM  involves  (1) 
decision  maker(s)  (2)  in  a  complex  environment  (3)  attempting  to  accomplish  one  or  more  tasks. 
Characteristics  of  each  of  these  three  constituent  parts  and  elaboration  of  the  key  elements  of  DDM 
are  discussed  in  the  upcoming  sections. 

3.1.1  Decision  Maker 

At  the  centre  of  any  dynamic  decision-making  event  is  the  decision  maker.  Decision-making  power 
can  be  centralized  to  one  person  or  may  be  distributed  to  many  people.  A  main  aim  of  dynamic 
decision  makers  is  to  gain  control  over  their  task  environment  (Clancy  et  ah,  2003).  This  is 
typically  done  through  making  a  series  of  decisions  (Brehmer  &  Allard,  1991)  rather  than  making  a 
single  choice.  The  key  issue  in  controlling  the  task,  moreover,  requires  the  decision  maker  to 
sustain  a  workable  compromise  between  the  demands  of  the  task  and  the  need  to  conserve  one’s 
cognitive  resources  (Brehmer,  2000).  To  gain  control,  successful  dynamic  decision  makers  must 
handle  two  types  of  problems  (Brehmer,  2000).  That  is,  dynamic  decision  makers  must  handle  the 
“core”  decision  task,  and  must  also  control  the  overall  decision  situation.  “Core”  decisions  are 
required  to  control  the  aspects  of  the  situation  that  are  of  concern  (Brehmer,  2000).  For  example, 
the  core  decision  task  in  military  command  and  control  is  to  defeat  the  adversary.  The  decision 
maker(s)  must  recognize  the  relationship  between  the  characteristics  of  the  controlling  process  and 
the  controlled  processes.  In  fire  fighting,  for  example,  the  fire  is  seen  as  a  process  with  temporal 
characteristics  (controlling  process)  and  the  mechanisms  available  to  the  fire  chief  to  fight  the  fire 
are  also  seen  as  a  process  (controlled  process).  The  tactical  methods  used  by  the  fire  chief  to  fight 
the  fire  are  based  on  the  relationship  between  these  two  processes.  Complicating  this  relationship  is 
the  time-dependent  nature  of  their  decisions  (e.g.,  it  takes  time  for  decisions  to  take  effect;  the 
magnitude  of  effect  varies  over  time).  Tactical  problems  are  incurred  when  the  controlled  process 
changes  faster  than  the  controlling  process  can  have  an  effect. 
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In  addition  to  handling  the  “core”  decision  tasks,  decision  makers  must  also  control  the  overall 
decision  situation  to  avoid  overload  and  to  remain  capable  of  making  core  decisions  (Brehmer, 
2000).  Given  the  complexity  of  the  situations  within  which  DDM  is  required,  becoming  distracted 
by  secondary  factors  could  lead  to  overload  and  the  inability  to  focus  on  the  most  critical  elements. 
A  key  part  of  controlling  the  decision  situation,  then,  may  involve  making  determinations  about 
how  to  balance  the  effects  of  core  decision  tasks  on  other  competing  goals.  In  fighting  a  fire,  for 
example,  Brehmer  (2000)  argues  that  the  fire  chiefs  core  task  is  to  control  the  fire.  On  the  other 
hand,  the  broad  situation  is  also  likely  to  introduce  a  number  of  secondary  demands,  such  as  the 
need  to  control  the  gathering  crowd,  or  the  need  to  coordinate  the  efforts  of  multiple  players 
working  to  diffuse  the  situation.  These  secondary  characteristics  of  the  situation  may  challenge  the 
fire  chiefs  ability  to  give  adequate  attention  to  the  most  critical  tasks. 

3.1.2  DDM  Tasks 

Dynamic  decision-making  occurs  across  a  broad  spectrum  of  tasks  including  fire  fighting,  military 
combat,  search  and  rescue,  and  medical  emergencies.  There  are  a  number  of  key  features  that 
characterize  dynamic  decision-making  tasks,  which  are  outlined  below. 

Interdependence:  DDM  tasks  are  highly  complex  because  they  are  typically  composed  of  multiple 
interdependent  components  that  can  influence  the  system  as  a  whole  as  well  as  each  other 
(Brehmer,  1990).  Earlier  decisions  can  constrain  later  decisions,  and  limit  the  ones  that  follow. 

Uncertainty.  DDM  tasks  are  highly  uncertain  for  a  number  of  reasons  (Clancy  et  al.,  2003). 
Uncertainty  can  be  due  to  the  fact  that  changes  in  the  DDM  system  occur  both  autonomously  and 
as  a  result  of  previous  decisions  or  actions  performed  in  the  system.  Another  factor  adding  to 
uncertainty  in  dynamic  decision  tasks  is  the  invisibility  of  some  aspects  of  the  system  (Gonzalez, 
2005).  This  opaqueness  (as  termed  by  Brehmer,  2000),  refers  to  a  “lack  of  transparency  about  the 
decision  situation”  (p.  239).  Opaqueness  can  result  from  a  lack  of  information  about  the  status  of 
the  DDM  task,  as  well  from  as  a  lack  of  information  regarding  the  relationship  between  the 
processes  to  be  controlled  (Brehmer,  2000;  Gonzalez,  Lerch,  &  Lebiere,  2003).  Feedbacks  delays 
are  an  important  cause  of  opaqueness. 

Time.  Time  is  a  key  element  of  dynamic  decision-making  (Brehmer  &  Allard,  1991)  because  DDM 
focuses  on  tasks  that  must  be  completed  in  real-time.  The  implication  of  this  is  that  the  decision 
maker  does  not  have  sole  control  over  when  decisions  need  to  be  made  (e.g.,  does  not  control  the 
pace  and  tempo  of  decisions).  As  Brehmer  ( 1 990,  p.  263)  argues,  “. .  .the  world  will  never  stop  and 
wait  for  him  to  make  his  decisions”.  Dynamic  decisions  sometimes  have  windows  of  opportunity 
within  which  actions  need  to  be  initiated  for  optimal  decisions  to  be  made.  Outside  of  that  window, 
the  environment,  situation,  and/or  task  requirements  can  change  and  the  decision  action  may  no 
longer  be  applicable.  Decision-actions  must  be  made  not  only  at  the  most  opportune  time,  but  also 
in  the  correct  order  (Brehmer,  1992).  Time  is  also  critical  in  DDM  because  each  task  may  be 
unique  in  terms  of  the  critical  time  scale  in  play.  When  fighting  a  large  fire,  for  example,  the  most 
immediate  time  problem  and  direct  effects  may  be  obvious,  but  other  factors  could  also  become 
time  critical. 

High  Levels  of  Risk.  There  is  an  element  of  risk  associated  with  decision  makers’  actions  (Clancy  et 
al.,  2003).  In  dynamic  decision-making  contexts,  the  stakes  and  the  cost  of  making  a  wrong 
decision  can  have  serious  consequences.  For  example,  a  firechief  who  sends  all  available  assets  to 
one  location  will  have  no  assets  available  to  send  out  should  another  emergency  arise  elsewhere 
(Brehmer,  2000). 
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Ill-Structured  Problems.  DDM  problems  are  rarely  well-defined  and  the  decision  maker  must 
struggle  to  identify  the  key  features  of  the  problem  (Clancy  et  ah,  2003).  This  can  be  difficult  as 
decision  makers  often  have  multiple  goals  and  the  priorities  of  these  goals  can  change  as  the 
situation  develops  (Flin,  1996;  as  cited  in  Clancy  et  ah,  2003). 

3.1.3  DDM  Environments 

When  trying  to  make  dynamic  decisions,  it  is  vital  to  understand  to  the  best  of  one’s  ability  the 
environment  in  which  the  decisions  are  made.  Brehmer  and  Allard  (1991)  suggest  that  dynamic 
decision  contexts  can  be  characterized  as  involving  complexity,  rate  of  change,  relations,  delays, 
feedback,  and  distribution  of  decision-making  capacity. 

Complexity.  DDM  environments  are  highly  complex.  Although  there  is  no  clear  operationalization 
of  complexity,  Brehmer  and  Allard  (1991)  define  the  complexity  in  a  given  situation  as  relative  to 
the  capacity  of  a  human  to  control  the  number  of  processes,  goals,  action  alternatives  and  side 
effects  of  the  system.  In  the  case  of  fire  fighting,  then,  complexity  would  emerge  as  a  product  of 
the  number  of  fires  in  play,  the  varying  goals  that  are  in  play  (i.e.,  some  fires  may  be  more  critical 
than  others),  the  number  of  resources  and  options  available,  and  the  secondary  effects  of  putting 
out  the  fires  (e.g.,  impact  on  the  environment).  Feedback  within  the  system  can  also  add  another 
source  of  complexity  (Hsiao  &  Richardson,  1999).  Positive  gains,  negative  feedback  loops,  and 
delayed  feedback  can  easily  be  mismanaged,  misinterpreted,  or  even  ignored,  which  adds  another 
layer  of  difficulty  to  already  cognitively  taxing  decision-making  (Hsiao  &  Richardson,  1999). 
Delays  in  any  link  of  the  chain  can  add  a  layer  of  complexity  that  requires  decision  makers  to 
account  for  lags  in  sending  and  receiving  information  and  initiating,  engaging,  and  completing 
decision-actions. 

Rate  of  change.  Rate  of  change  refers  to  how  quickly  the  processes  to  be  controlled  change. 
Changes  can  be  very  slow  (e.g.,  controlling  a  country’s  economy)  or  very  fast  (e.g.,  performing  a 
low-level  fighter  jet  attack). 

Relation  between  the  characteristics  of  the  process  to  be  controlled  and  those  of  the  control 
processes.  DDM  requires  the  decision  maker  to  control  a  time-dependent  process.  However,  the 
means  to  control  this  process  are  also  dynamic  (e.g.,  it  takes  time  for  decisions  to  take  effect;  the 
magnitude  of  effect  varies  over  time).  If  the  process  under  control  changes,  then  the  effectiveness 
of  the  actions  used  to  control  the  process  will  also  change.  Brehmer  and  Allard  (1991)  again  use  a 
fire  fighting  example  -  if  the  delay  in  getting  fire  fighting  units  (FFU)  to  the  scene  is  not  figured 
into  the  estimate  of  how  many  units  will  be  required,  the  fire  could  be  larger  when  these  FFU 
actually  reach  the  fire,  making  it  impossible  to  control  the  situation. 

Feedback  Delays.  Delays  refer  to  when  the  transmission  of  information  or  energy  slows  down  or 
lags  behind  in  the  dynamic  decision  system.  Delays  can  be  quite  complex  because  they  may  occur 
in  different  locations  in  the  system.  Within  a  fire  fighting  team,  for  example,  delays  could  be 
experienced  getting  commands  to  the  FFUs,  in  the  FFUs  actually  responding  to  the  delays. 

Quality  of  feedback  information.  The  quality  of  the  information  about  the  progress  of  the  task  can 
vary,  which  can  be  a  source  of  uncertainty  for  dynamic  decision  makers.  This  variance  can  be  a 
product  of  the  information  systems  used  to  transmit  information,  or  of  the  quality  of  the  reports 
sent  by  other  team  members. 

Distribution  of  decision-making  capacity.  Decision-making  power  can  be  centralized  to  one  person 
or  be  distributed  to  many  people.  How  decision-making  capacity  is  distributed  is  likely  to  play  a 
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key  role  in  how  DDM  occurs.  If  the  working  “model”  of  the  system  does  not  incorporate  the  delays 
inherent  within  a  centralized  system,  this  model  will  not  be  helpful  because  it  will  be  out  of  date. 

3.2  Domain  related  perspectives  on  DDM 

DDM  has  been  approached  and  studied  from  a  number  of  different  perspectives.  These 
perspectives  include  system  dynamics,  psychology  and  engineering  science.  Each  of  these 
perspectives  views  DDM  uniquely  and  emphasizes  distinct  aspects  of  the  DDM  process.  Examples 
of  these  perspectives  are  discussed  in  more  detail  in  the  sections  that  follow. 

3.2.1  System  Dynamics:  Stock  &  Flow 

The  system  dynamics  approach  aims  to  understand  the  behaviour  of  complex  systems  over  time 
(http://en.wikipedia.org/wiki/System_dynamics).  Systems  theory  is  often  distinguished  by  its  focus 
on  nonlinearity,  and  its  use  of  internal  feedback  loops,  time  delays  that  affect  the  entire  system,  as 
well  as  the  use  of  stocks  and  flows.  Several  thinking  skills  have  also  been  identified  as  critical 
within  a  systems  perspective  (Sweeney  and  Sterman,  2000),  including  the  ability  to: 

•  Understand  how  behavior  of  the  system  arises  from  the  integration  of  its  agents  over 
time  (i.e.,  dynamic  complexity); 

•  Discover  and  represent  feedback  processes  (positive  and  negative)  hypothesized  to 
underlie  observed  patterns  of  system  behavior; 

•  Identify  stock  and  flow  relationships; 

•  Recognize  delays  and  understand  their  impact; 

•  Identify  nonlinearities;  and 

•  Recognize  and  challenge  the  boundaries  of  mental  (and  formal)  models. 

According  to  Elg  (1996),  system  dynamics  offers  a  framework  for  understanding  DDM  and 
provides  ideas  on  how  to  improve  learning  in  and  about  complex  dynamic  systems.  System 
dynamics  provides  a  way  for  us  to  improve  our  understanding  of  the  systems  we  want  to  control. 
Given  the  complexity  and  emphasis  on  “flow”  in  DDM,  then,  it  is  perhaps  unsurprising  that 
systems  theory  has  been  frequently  used  to  understand  and  explore  DDM. 

System  dynamics  has  two  general  goals  (Elg,  1996).  The  first  goal  is  to  understand  complex, 
dynamic  systems  by  modeling  and  analyzing  the  system.  This  is  achieved  by  conducting  empirical 
and  theoretical  studies  on  real  life  problems  and  implementing  the  results  of  the  studies  in  a 
simulation  model  for  analysis.  The  second  goal  is  to  improve  system  reasoning  abilities  and  to 
develop  the  ability  to  understand,  conceptualize  and  build  models  of  systems.  In  DDM  tasks,  these 
goals  help  researchers  understand  and  improve  DDM  behaviours  from  a  holistic,  systemic 
perspective. 

A  system  dynamics  approach  to  measuring  DDM  often  uses  stock  and  flow  scenarios  as 
frameworks  for  investigating  complex  systems.  Many  stock  and  flow  scenarios  will  be  presented  in 
this  section  in  the  form  of  a  supply  chain  system  microworlds  as  used  by  Cronin  and  Gonzalez 
(2007),  Fu  and  Gonzalez  (2006),  and  Sterman  and  Diehl  (1993)  (for  more  on  Microworlds  see 
section  3.3). 
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Sterman  (2002;  as  cited  in  Cronin  &  Gonzalez,  2007)  suggests  that  people  have  problems 
understanding  system  dynamics.  It  is  suggested  that  such  difficulties  are  because  we  have  a  poor 
understanding  of  the  building  blocks  of  system  dynamics,  including  stocks,  flows  and  time  delays 
(Cronin  and  Gonzalez,  2007).  Cronin  and  Gonzalez  (2007)  were  interested  in  understanding  the 
cognitive  functions  that  explain  why  people  misunderstand  the  relationship  between  stocks  and 
flows.  They  conducted  a  series  of  three  studies  to  test  whether  the  familiarity  of  the  dynamic 
system,  cognitive  effort,  computational  difficulty,  and/or  graphical  features  were  responsible. 

The  first  study  explored  possible  reasons  why  participants  in  previous  research  had  difficulty 
understanding  stocks  and  flows.  They  looked  at  two  possibilities:  1)  the  cover  stories  of  previous 
studies  did  not  highlight  the  “stocks”  portion  of  the  model  resulting  in  an  incorrect  mental  model  of 
the  system,  and  2)  participants  in  previous  studies  used  little  effort  to  think  about  the  problem.  The 
study  asked  university  participants  to  look  at  different  graphs  and  describe  the  amounts  and 
movements  of  stocks.  The  researchers  found  no  significant  differences  between  groups  as  the  result 
of  two  different  cover  stories  (one  that  highlighted  the  stocks  portion  of  the  model  and  one  that  did 
not)  or  as  a  product  of  varying  levels  of  thinking  effort.  This  finding,  Cronin  and  Gonzalez  argue, 
lends  support  to  the  idea  that  people  may  not  develop  sufficient  mental  models  or  heuristics 
necessary  to  adequately  support  DDM  in  a  dynamic  stock  and  flow  environment. 

The  next  two  studies  investigated  the  role  of  visual  form  in  constructing  inappropriate 
representations  of  problems  that  may  lead  to  poor  performance.  Participants  were  asked  to  use 
information  from  a  number  of  visually  different  graphs.  The  results  of  these  studies  led  Cronin  and 
Gonzalez  (2007)  to  conclude  that  the  visual  representation  of  dynamic  systems  can  influence 
people’s  understanding  of  the  relationship  between  stocks  and  flows.  That  is,  graphical  depictions 
direct  attention  to  some  things  but  not  others.  This  suggests  that  the  visual  representation  of  the 
dynamic  system,  or  how  the  decision  maker  interprets  the  system  representation,  may  have  a 
significant  role  in  the  design  of  the  heuristics  and  mental  models  used  in  DDM. 

Fu  and  Gonzalez  (2006)  propose  that  heuristics  used  may  produce  stable  behaviour  in  one  setting 
and  oscillation  in  another  solely  as  a  function  of  the  feedback  structure  in  which  it  is  embedded. 
They  suggest  that  heuristics  may  not  be  adequate  to  support  the  DDM  task.  In  some  cases  where 
people  may  be  successful  in  controlling  the  system,  success  may  be  a  function  of  an  incidental 
match  between  the  decision  maker’s  behaviour  and  what  the  system  requires,  rather  than  a 
successful  match  of  conscious  effort  on  the  part  of  the  decision  maker  (Fu  &  Gonzalez,  2006). 

In  order  to  investigate  this,  Fu  and  Gonzalez  (2006)  conducted  research  exploring  two  questions 
related  to  learning  of  temporal  dynamics  in  dynamic  decision-making.  The  first  question  related  to 
what  information  is  utilized  and  how  this  information  changes  with  experience.  The  second 
question  explored  what  different  strategies  are  dominant  when  comparing  learning  behaviour  in 
static  versus  dynamic  decision-making  situations. 

Fu  and  Gonzalez  (2006)  used  a  simplified  supply  chain  system  in  a  microworld  platform  called  The 
Beer  Game.  In  this  game,  a  single  retailer  supplies  beer  to  the  consumer,  a  single  wholesaler 
supplies  beer  to  the  retailer,  the  distributor  supplies  beer  to  the  wholesaler,  and  the  factory  to  brews 
and  supplies  beer  to  distributor.  The  object  of  the  game  is  to  minimize  inventory  and  avoid 
backorder  while  maximizing  profit.  Delays  are  introduced  into  the  system  in  order  to  manipulate 
the  complexity  of  the  system  and  encourage  changes  in  the  strategies  used  to  make  decisions  in  this 
dynamic  environment. 

Fu  and  Gonzalez  (2006)  showed  that  initially  participants  failed  to  sufficiently  change  their 
decision-making  strategies.  They  tended  to  ignore  temporal  dynamics  of  the  system  and 
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consequently  underutilize  information  that  indirectly  influenced  the  outcome  of  decisions  (Fu  & 
Gonzalez,  2006).  With  practice,  however,  participants  learned  to  use  relevant  supply  line 
information  to  anticipate  customer  demand  and  learn  to  ignore  irrelevant  information.  This  learning 
effect  was  not  found  by  Sterman  and  Diehl  (1993),  who  also  used  a  systems  dynamic  approach  to 
investigate  the  effects  of  time  delays  and  feedback  processes  on  DDM  performance. 

Using  a  microenvironment  of  a  managing  firm,  Sterman  and  Diehl  (1993)  found  no  learning  effects 
based  on  the  time  taken  to  act  on  the  system  when  a  decision-action  was  required.  Two  main 
sources  of  underperformance  were  identified.  First,  people  would  employ  the  correct  model,  but 
apply  it  inconsistently.  This  mistake  is  fueled  by  the  failure  to  properly  account  for  the  time  delays 
or  feedback  loops  (Sterman  &  Diehl,  1993).  Second,  an  incorrect  model  was  consistently 
employed.  Specifically,  it  seemed  that  participants  did  not  properly  account  for  the  importance  of 
relevant  information  (mainly  underweighting  of  inventory,  future  production,  and  stages  of  the 
supply  line)  (Sterman  &  Diehl,  1993). 

Though  incorrect  models  may  be  employed,  it  was  demonstrated  by  Fu  and  Gonzalez  (2006)  that 
people  do  possess  the  cognitive  machinery  necessary  to  deal  with  DDM  tasks.  And,  with  extended 
practice  and  coaching,  people  are  capable  of  controlling  complex,  dynamic  systems.  Fu  and 
Gonzalez  (2006)  provided  more  evidence  that  the  relevance  of  information  directly  impacts 
performance.  When  participants  were  presented  with  only  the  relevant  and  necessary  information 
within  the  system,  performance  was  better  compared  to  those  who  were  provided  with  both 
relevant  and  irrelevant  information  where  no  distinction  was  made  between  the  two.  This  suggests 
that  the  absence  of  irrelevant  information  helps  participants  to  learn  temporal  dynamics  of  system 
and  result  in  better  performance  when  controlling  DDM  in  a  microworld  platform  in  stock  and  flow 
environments. 

Results  from  both  studies  suggest  that,  at  least  initially,  people  have  trouble  dealing  with  long  time 
delays  between  actions  and  feedback  (in  supply  line)  (Fu  &  Gonzalez,  2006;  Sterman  &  Diehl, 
1993).  This  supports  the  misperception  of  feedback  (MOF)  hypothesis  defined  by  Sterman  &  Diehl 
(1993)  that  states  people  have  an  ‘open-loop’  view  of  causality;  they  fail  to  account  for  delays 
between  action  and  response,  and  acquisition  of  information;  ignore  feedback  processes;  do  not 
sufficiently  understand  stocks  and  flows;  and  are  “insensitive  to  nonlinear  complexities  in  the 
system  that  may  change  the  relevance  of  different  feedback  loops  as  a  system  evolves”  (p.  1). 

The  studies  explored  in  this  section  show  how  a  systems  dynamic  approach  allows  dynamic 
decision-making  to  be  experimentally  manipulated  in  realistic  real-world  stock  and  flow 
environments.  System  dynamics  research  has  helped  to  clarify  and  understand  some  of  the 
strategies  and  processes  people  use  when  faced  with  complex  situations  involving  different  sources 
of  visual  representation  of  the  system  (Cronin  &  Gonzalez,  2007);  delayed  feedback  (Sterman  & 
Diehl,  2003;  Fu  &  Gonzalez,  2006);  varying  amounts  and  types  of  relevant  or  irrelevant 
information  (Fu  &  Gonzalez,  2006);  and  sources  of  underperformance  (Sterman  &  Diehl,  2003). 
Systems  dynamics  research  may  be  particularly  useful  in  trying  to  identify  mental  models  and 
heuristics  used  in  DDM  as  there  has  been  a  substantial  “. .  .portion  of  [system  dynamics]  research 
effort  to  developing  a  wide  variety  of  techniques  and  procedures  for  eliciting,  representing,  and 
mapping  mental  models  to  aid  model  building’  (Flail,  Aitchison  &  Kocay,  1994;  as  cited  in  Doyle 
&  Ford,  1998,  p.3).  An  assumption  that  permeates  throughout  system  dynamics  is  that  mental 
models  are  created,  modified  and  used  to  mediate  all  DDM.  A  major  problem  is  that  definitions  of 
mental  models  are  typically  general,  vague,  and  authors  often  disagree  (Doyle  &  Ford,  1998). 

The  system  dynamics  approach  to  DDM  has  limitations  that  confine  the  study  of  DDM  to  contexts 
where  it  is  assumed  people  have  a  general  knowledge  and  understanding  of  the  system,  its 
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components  (e.g.,  key  information  about  demands  and  supply  is  known),  its  structure,  and  function; 
the  ability  to  understand,  analyze,  and  create  mental  models  about  the  system;  and  that  these 
assumptions  are  influenced  by  reasoning  ability  and  have  important  impact  on  DDM  performance. 
Though  this  approach  is  able  to  provide  a  method  of  understanding  and  experimentally  examining 
DDM  in  certain  environments,  this  approach  may  not  be  equally  applicable  to  the  wide  range  of 
real-world  scenarios  where  DDM  is  found. 

3.2.2  Psychology  -  DDM  as  Choice 

Psychology  has  also  been  active  in  recent  decades  working  to  understand  dynamic  decision¬ 
making.  It  is  important  to  note  that  this  has  not  always  been  the  case,  however.  In  fact,  as  late  as 
1990,  Brehmer  (1990,  p.  264)  argued  that  even  though  examples  of  “real-life,  dynamic  tasks  are 
not  hard  to  find”,  he  lamented  that  these  tasks  “. .  .have  received  little  attention  from  psychologists, 
except  in  the  form  of  a  few  scattered  studies  of  process-control  tasks”  (Brehmer,  1990,  p.  264). 
Brehmer  (2000)  argued  that  there  were  two  possible  reasons  for  psychology’s  relative  neglect  of 
DDM.  First,  he  argued  that  the  research  methodologies  needed  to  explore  it  were  lacking,  as 
traditional  psychological  research  relied  on  paper-and-pencil  approaches.  Second,  he  argued  that 
the  normative  models  typically  used  in  behaviour  decision  research  (e.g.,  Slovic,  Fischoff  & 
Lichtenstein,  1977)  could  not  adequately  capture  dynamic  decision-making. 

These  analytic  theories  of  decision-making  emphasized  normative  and  rational  approaches,  which 
tended  to  argue  that  people  proceed  progressively  through  a  rational  process  of  problem-solving 
when  presented  with  issues  in  complex  situations.  According  to  Brehmer  and  Allard  (1991,  p.320), 
“a  decision  problem  is  defined  in  terms  of  a  set  of  possible  actions,  connected  to  a  set  of  possible 
consequences  of  these  actions  by  means  of  probabilities.  The  decision  maker’s  task  is  to  select  the 
option  that  leads  to  the  best  outcome,  usually  defined  as  the  alternative  with  the  highest  expected 
value.  Thus,  decision-making  is  seen  as  a  matter  of  resolving  a  choice  dilemma.”  Indeed,  these 
types  of  approaches  to  understanding  decision-making  have  often  portrayed  it  as  providing  “an 
array  of  well-specified  alternatives  on  the  basis  of  some  form  of  a)  subjective  expected  probability 
or  b)  a  utility  maximized  algorithm”  (Clancy  et  ah,  2003,  p.  588).  This  research  has  also  often  used 
highly  artificial  tasks  and  has  often  involved  trivial  rather  than  consequential  outcomes  as  the  result 
of  the  decision-making  process  (Clancy  et  ah,  2003). 

As  decision-making  theory  progressed,  however,  it  began  to  emphasize  more  complex 
environments,  and  extended  beyond  the  need  to  make  a  single  “maximal  return”  choice  to  the  need 
to  control  multiple  sequences  of  interrelated  decisions.  The  evolving  descriptions  of  decision¬ 
making  focused  not  on  how  people  should  make  complex  decisions,  but  on  how  they  actually  did 
make  these  decisions  in  the  real  world.  In  keeping  with  this  trend  away  from  the  purely  rational 
forms  of  decision-making,  a  considerable  body  of  work  has  explored  more  naturalistic  and  intuitive 
forms  of  decision-making. 

Several  different  lines  of  research  have  generated  an  interest  in  more  intuitive  models  of 
conventional  decision-making  as  an  alternative  to  the  strict  rational  models.  Known  generally  as 
Naturalistic  Decision-making  (NDM),  this  approach  has  been  defined  as  the  study  of  how  people 
use  their  experience  to  make  decisions  in  the  real  world  (Zsambok  &  Klein,  1997;  cited  in  Pliske  & 
Klein,  2003;  Bakken  &  Vamraak,  2003),  often  under  time  pressure,  risk,  and  uncertainty.  NDM 
diverges  from  more  traditional  approaches  of  decision-making  because  it  strives  to  consider 
decisions  in  context  rich  settings,  people  with  domain  experience,  descriptions  of  decision-making 
strategies,  and  pre-choice  processes  such  as  the  development  of  situation  awareness  (Zsambok, 
1997;  cited  in  Pliske  &  Klein,  2003).  In  situations  of  limited  time,  high  risk  and  a  great  deal  of 
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uncertainty,  searching  for  the  optimum  solution  to  a  difficult  decision  (as  prescribed  by  normative 
models  of  decision-making)  might  actually  hinder  the  process  as  opposed  to  improving  it.  This 
work  emphasizes  the  notion  that  decisions  (particularly  those  in  complex  situations)  are  typically 
not  made  through  the  careful  sifting  and  rational  weighting  of  alternatives,  but  that  factors  such  as 
accumulated  expertise  and  pattern  matching  help  to  guide  intuitive  decisions.  Accepting  that  sub- 
optimal  solutions  may  have  benefits  is  in  contrast  to  the  system  dynamics  perspective  that  preferred 
to  always  compare  dynamic  decisions  to  the  optimal  solution. 

This  suggests  that  more  complex  models  of  decision-making  might  be  necessary  to  capture 
complex  and  dynamic  processes.  One  issue  that  might  promote  effectiveness  would  be  experience 
in  the  target  domain.  In  military  operations,  for  example,  commanders  are  continuously  receiving 
status  information  and,  on  the  basis  of  that  information,  managing  their  resources.  Once  the 
directives  are  acted  upon,  subordinates  will  report  back  with  outcomes  and  new  status  report.  This 
cycle  continues  for  the  duration  of  the  operation.  The  cyclical  nature  of  military  operations 
suggests  that  military  planners  would  be  better  at  making  intuitive  judgments  than  most  decision 
makers.  However,  there  is  also  evidence  that  even  more  complex  models  may  not  necessarily  be 
wholly  adequate.  Baaken  and  Vamraak  (2003)  hypothesized  that  even  military  planners  would  face 
the  same  challenges  as  other  people  making  intuitive  judgments.  To  test  this  hypothesis,  they  had 
seven  highly  educated  defence  analysts  conduct  a  “Peace  Support”  planning  task  focusing  on  the 
dynamic  interrelations  between  budget  allocations  and  cost  of  operations  (initial  purchases  and 
running  sustenance).  Included  in  the  scenario  was  also  a  one-time  budget  increase  that  could  lead 
to  a  short-term  performance  boost  as  well  as  a  long-term  penalty.  Each  participant  was  provided 
with  a  task  description  and  an  answer  grid.  They  were  asked  to  complete  the  answer  grid  with  a 
hand  drawn  sketch  of  the  estimated  total  effect  curve  between  the  time  of  starting  extra  funding  and 
40  days  ahead  in  the  simulation. 

Baaken  and  Vamraak  (2003)  found  that  the  participants  were  consistently  over-optimistic  and 
demonstrated  only  a  partial  understanding  of  the  dynamics  generated  by  the  logistics  chain 
structure.  Participants  were  unable  to  reproduce  the  cyclic  performance  resulting  from  the  budget 
increase.  The  hand  drawn  graphs  resembled  a  “plain”  budget  profde  stretched  out  in  time. 
Subsequently,  Bakken  and  Vamraak  (2003)  concluded  that  intuition  alone  is  not  always  an 
adequate  guide  for  dynamic  decision-making. 

Other  psychological  approaches  to  understanding  DDM  have  focused  on  cognitive  processes  such 
as  learning,  planning,  and  problem-solving.  Busemeyer  (1999)  suggests  that  learning  processes 
may  account  for  much  of  the  variance  seen  in  human  performance  on  dynamic  decision-making 
tasks.  For  example,  instance  or  exemplar-based  models  assume  that  actions  leading  to  successful 
outcomes  are  stored  together  in  memory,  to  be  retrieved  when  the  decision  maker  encounters 
similar  situations.  Gonzalez  et  al.  (2003)  propose  a  learning  theory  of  DDM  called  “instance  based 
learning  theory”  (IBLT).  IBLT  proposes  that  people  have  instance-action  based  constructs  that  are 
stored  and  linked  together  in  memory.  These  coupled  constructs  are  retrieved  on  the  basis  of 
similarity  to  a  current  situation.  When  the  instance  is  matched,  the  behaviour  is  recognized  as  a 
viable  action  to  be  performed  in  the  current  situation.  As  the  instance  based  repertoire  is 
accumulated,  stored,  and  improved  upon  with  experience,  learning  can  take  place.  In  DDM  people 
learn  by  accumulation,  recognition,  and  refinement  of  instances  (information  about  the  decision¬ 
making  situation,  action,  result  of  decision)  (Gonzalez  et  al.,  2003) 

A  variety  of  DDM  research  reveals  much  about  the  flaws  in  people’s  ability  to  manage  dynamic 
complexity  (Sterman  &  Diehl,  1993).  Sterman  (1987)  conducted  a  study  that  ran  participants 
through  a  DDM  experiment  and  then  modeled  their  responses  based  on  two  known  heuristics  (i.e.. 
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anchoring  and  adjustment  heuristics).  The  system  chosen  for  the  experiment  was  called  the 
multiplier-accelerator  model  of  capital  investment.  Participants  in  this  study  included  both  students 
and  economic  professionals  knowledgeable  about  economic  and  control  theory.  Despite  only  a 
single  simple  system  input  change,  only  8%  of  the  participants  were  able  to  re-establish 
equilibrium  before  the  end  of  the  simulation.  A  model  of  the  decision  responses  based  on  the 
anchoring  and  adjustment  heuristic  proved  to  effectively  mimic  the  DDM  patterns  of  participants. 
This  is  consistent  with  Elg  (1996)  who  noted  that  when  identifying  problems,  people  conceptualize 
their  environments  and  make  conclusions  according  to  their  perceived  state  of  the  situation. 

After  further  studies  showing  consistency  in  participants’  sub-optimal  performance,  Sterman  and 
Diehl  (1993)  postulate  possible  causes.  They  attribute  this  poor  performance  to  two  possible 
reasons.  First,  they  argue  that  participants  may  be  unsuccessful  due  to  the  fact  that  they  do  not 
understand  the  system  because  of  its  complexity,  and  thus  use  minimum  effort  to  maintain  a 
minimum  level  of  control.  Second,  they  argue  that  participants  may  suffer  from  two  types  of 
misperceptions  of  feedback.  The  first  misperception  may  be  that  that  people  are  insufficiently 
aware  of  the  structure  of  the  feedback  system.  Because  of  this,  they  tend  to  use  a  ‘hands-off 
strategy  failing  to  understand  how  their  actions  influence  the  system.  The  second  feedback  problem 
is  that  people  are  unable  to  account  for  delays  and  feedback  effects.  This  could  the  result  of  either 
faulty  mental  models  that  are  oversimplified  for  complex  control  tasks,  or  because  people  have  a 
poor  ability  to  correctly  infer  the  behaviors  necessary  to  control  complex  feedback  systems. 

In  general,  however,  it  is  very  difficult  to  truly  disentangle  conceptualizations  of  DDM  from  the 
psychological  domain  with  those  from  the  system  dynamics  domain,  and  there  seems  to  be  a  good 
deal  of  overlap  in  these  approaches  to  DDM.  Perhaps  the  best  way  to  distinguish  them  is  by 
examining  what  processes  they  emphasize.  From  the  psychological  perspective,  for  example, 
processes  such  as  decision-making,  learning,  planning  and  problem  solving  are  clearly  relevant  to 
DDM.  However,  from  the  systems  perspective,  the  focus  is  not  on  these  psychological  processes 
per  se,  but  on  the  instantiation  of  these  processes  using  system  analogies  (e.g.,  stock  and  flow 
diagrams).  From  the  psychological  perspective,  then,  the  choices  of  the  individual  within  the  DDM 
situation  may  be  the  focus,  whereas  the  system  analogy  aims  to  instantiate  the  mental  process  in  an 
observable  way. 

3.2.3  Control  Theory  -  DDM  as  Control 

In  a  1 990  article  by  Brehmer,  it  is  argued  that  conventional  psychological  approaches  may  be 
inadequate  for  capturing  DDM,  in  part  because  these  approaches  had  been  based  in  the  normative 
models  used  in  behavioural  decision  research.  In  attempting  to  find  another  framework  for 
conceptualizing  DDM,  Brehmer  (1990)  argues  that  control  theory  might  be  a  helpful  approach. 

Control  theory  is  a  branch  of  engineering  that  deals  with  the  behaviour  of  dynamic  systems,  and 
mathematically  models  the  ways  one  process  controls  another  process  (Brehmer,  2000).  Given  that 
the  objective  of  dynamic  decision-making  is  to  achieve  control  over  some  aspect  of  the 
environment  (i.e.,  decisions  are  made  to  achieve  a  desired  state  of  affairs  or  to  keep  a  system  in  a 
desired  state),  Brehmer  (1992,  2000;  Brehmer  &  Allard,  1991)  suggested  that  control  theory  could 
be  used  as  a  framework  for  guiding  DDM  research.  From  a  psychological  perspective,  DDM  is 
defined  in  terms  of  resolving  a  choice  dilemma,  whereas  from  a  control  theory  view,  DDM  is 
defined  as  “. .  .the  process  of  achieving  control  over  a  system  in  order  to  produce  a  desired 
outcome”  (Brehmer,  1990,  p.  265).  That  is,  control  theory  was  argued  to  be  helpful  in 
understanding  DDM  because  it  specifies  the  general  conditions  required  to  control  a  system.  These 
general  conditions  are  that: 
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•  There  must  be  a  goal  (goal  condition). 

•  It  must  be  possible  to  ascertain  the  state  of  the  system  to  be  controlled  (observability 
condition). 

•  It  must  be  possible  to  change  the  state  of  the  system  (change  condition). 

•  There  must  be  a  model  of  the  system  that  describes  what  will  happen  if  changes  are  made 
to  the  system  (model  condition). 

This  model  condition  is  particularly  critical  to  this  review,  because  it  argues  that  “. .  ..to  control  a 
system,  a  control  device,  such  as  a  decision  maker,  must  have  (or  be)  a  model  of  the  system  it  seeks 
to  control”  (Brehmer,  1990,  p.  265).  This  requirement,  of  course,  is  the  reason  that  understanding 
the  mental  models  that  people  develop  as  they  work  to  control  dynamic  systems  has  been  seen  as  a 
critical  task  for  control  theory  researchers. 

From  the  perspective  of  control  theory,  there  are  two  basic  control  strategies  (Brehmer,  1990). 
These  include  feedforward  control  and  feedback  control.  Feedforward  control  relies  on  predictions 
about  the  future  state  of  the  systems,  and  feedback  control  involves  choices  based  on  current 
system  information.  Moreover,  these  distinct  strategies  require  somewhat  different  models,  and 
have  different  requirements  for  optimizing  control  over  the  system. 

In  the  case  of  feedforward  strategies,  then,  they  require  that  the  system  does  not  change  over  time. 
And,  in  building  a  model  of  the  DDM  situation,  the  decision  maker  is  typically  one  of  many  actors 
in  the  situation.  This  person  must  not  only  “work  ahead”  and  be  proactive,  but  must  also  figure  the 
time  requirements  of  the  relevant  processes  into  the  working  model  of  the  situation.  Expecting  and 
building  in  delays  will  help  to  ensure  that  one’s  model  of  the  situation  is  as  accurate  as  possible. 
Another  way  to  manage  inevitable  delays,  Brehmer  (1990)  argues,  is  to  decentralize  the  decision¬ 
making  process  such  that  there  is  less  need  for  coordination. 

For  feedback  strategies,  however,  they  only  need  to  consider  how  previous  control  actions  have 
affected  the  system;  hence,  they  can  be  much  simpler.  If  accurate  feedback  information  can  be 
received,  the  biggest  potential  challenge  is  the  magnitude  of  the  delays  in  relation  to  the  time 
needed  to  “correct”  the  system.  If  these  delays  are  large,  the  situation  could  easily  spiral  out  of 
control  without  the  time  for  corrective  action  to  be  introduced.  Hence,  feedback  strategies  alone  are 
not  typically  adequate  for  optimal  DDM.  Some  studies  have  shown  feedback  control  to  be  less 
effective  compared  to  feedforward  control  in  DDM  (Schultz,  Dutta,  &  Johnson,  2000). 

Interestingly,  Brehmer  ( 1 990)  also  argues  that  feedback  strategies  are  inherently  more  likely  to  be 
used  than  feedforward  strategies.  They  require  simpler  models,  and  they  may  be  successful  (even 
though  suboptimal)  and  any  apparent  success  may  stifle  the  search  for  more  proactive  (and  perhaps 
more  successful)  strategies.  In  a  sense,  this  description  of  the  dynamic  decision  maker  argues  that 
people  may  be  inclined  to  “satisfice”  on  a  non-optimal  solution  because  it  requires  a  less 
demanding  model  of  the  situation.  This  use  of  simpler  mental  models,  moreover,  is  also  argued  to 
be  one  of  the  reasons  why  researchers  have  had  difficulty  capturing  mental  models.  Brehmer 
(1990,  p.  267-268)  argues 

“...we  cannot  always  expect  the  decision  maker  to  have  a  well-developed  model  of  the 
task.  If  decision  makers  follow  a  feedback-control  strategy,  the  need  for  such  models  will 
not  be  obvious  to  them.  Although  they  will  need  some  model  of  how  different  actions  will 
affect  the  system,  this  is  not  the  same  as  having  a  general  model  of  the  task.  Therefore,  if 
our  subjects  are  unable  to  answer  questions  about  the  nature  of  the  task  system,  this  cannot 
be  interpreted  as  evidence  that  they  are  unaware  of  their  model;  it  may  simply  mean  that 
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we  have  probed  for  the  wrong  model,  that  is,  that  we  have  asked  for  information  pertaining 
to  feedforward  control  when  the  subjects  are  following  a  feedback  strategy.” 

The  key  focus  in  control  theory,  then,  involves  understanding  the  strategies  that  people  use,  and 
their  models  of  the  relevant  task.  The  key  issues  relate  to  how  the  decision  maker  uses  feedback 
from  the  environment,  and  whether  the  model  is  more  complex  (in  the  case  of  a  feedforward 
strategy)  or  simple  (in  the  case  of  feedback  strategies).  In  opposition  to  rational  models  of  human 
decision-making,  then,  where  the  decision-maker  is  making  discrete  choices,  control  theory  focuses 
on  the  “continuous  flow  of  behaviour  and  monitoring  one’s  progress  toward  some  goal  rather  than 
discrete  episodes  involving  choice  dilemmas”  (Brehmer,  1990,  p.  268). 

It  is  important  to  note,  however,  that  Brehmer’ s  use  of  control  theory  to  understand  DDM  is 
intended  to  be  used  in  a  “loose  metaphorical  way”  (Brehmer,  1991,  p.  268)  rather  than  formally, 
because  “current  forms  of  control  theory  are  not  really  congruent  with  what  we  know  about  human 
behaviour”.  Control  theory  requires  input  to  be  described  in  terms  of  measured  individual  signals, 
whereas  human  perception  is  based  on  patterns  and  gestalts.  Because  of  the  importance  of  patterns 
and  gestalts,  then,  mathematical  equations  may  be  of  little  use  for  modelling  human  decision¬ 
making  (Brehmer,  1990).  Acknowledging  the  restrictions  associated  with  using  control  theory  as  a 
framework  for  DDM  research,  Brehmer  (1992;  2000)  argues  that  control  theory  be  used  as  a 
metaphor  for  DDM. 

Gibson,  Fichman  and  Plant  (1997;  as  cited  in  Gonzalez,  2005)  have  built  on  Brehmer’ s  (1992) 
application  of  control  theory  to  understand  DDM.  Gibson  et  al.  proposed  a  model  of  DDM  learning 
based  on  control  theory.  Their  learning  model  describes  decision-making  in  terms  of  two 
submodels:  the  judgement  submodel  and  the  choice  submodel.  The  judgment  submodel  states  that 
people  learn  by  minimizing  the  difference  between  predicted  outcomes  and  actual  outcomes, 
whereas  the  choice  submodel  states  that  people  learn  by  minimizing  the  differences  between 
choices  predicted  by  the  judgment  submodel  and  the  choices  they  actually  made.  The  Gibson  et  al. 
learning  model  of  DDM  has  been  implemented  computationally  via  neural  networks  and  has  been 
found  to  provide  a  good  account  of  learning  in  dynamic  situations  as  well  as  how  knowledge  is 
transferred  in  novel  situations  (Gibson,  2000;  as  cited  in  Gonzalez,  2005).  Therefore,  it  can  be  seen 
that  control  theory  is  a  useful  way  to  help  understand  DDM  even  if  it  can  not  be  used  directly  as  a 
theoretical  model  of  DDM. 

3.3  Microworlds 

According  to  Brehmer  (1992),  a  key  component  of  making  correct  decisions  is  having  an  accurate 
model  of  the  decision  situation.  When  the  situation  and  the  model  of  the  situation  differ,  errors  are 
inevitable.  One  way  of  developing  more  accurate  measures  of  people’s  learning  and  decision¬ 
making  is  by  developing  more  accurate  decision-making  models  through  the  use  of  microworlds 
(Elg,  1996).  A  microworld  is  a  complex  computer  simulation  used  in  controlled  experiments 
designed  to  study  dynamic  decision-making  (Gonzalez,  Vanyukov,  &  Martin,  2005).  Microworlds 
have  been  used  to  research  DDM  since  the  1970s  in  Australia,  Germany,  the  US,  Sweden  and 
Canada  (Brehmer,  2000;  Jarmasz,  2006). 

Microworlds  as  a  research  tool  are  a  compromise  between  the  experimental  control  of  laboratory 
research  and  the  realism  of  field  research  (Brehmer  &  Ddmer,  1993).  Although  microworlds  are 
relatively  simple,  they  embody  the  essential  characteristics  of  real-world  DDM  environments  (e.g., 
complexity,  opaqueness).  Simulations  can  represent  the  structure  and  complexity  of  aggregate 
dynamic  systems  with  great  fidelity  and  permit  controlled  manipulations  of  the  decision  contexts  in 
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the  system  (Sterman,  1989;  as  cited  in  Elg,  1996).  Therefore,  microworlds  provide  the 
experimental  control  needed  to  develop  explanations  of  DDM  (Gonzalez,  Vanyukov  et  ah,  2005). 

Researchers  using  micro  worlds  have  typically  used  two  different  strategies:  the  individual 
differences  strategy  and  the  experimental  strategy  (Brehmer,  2000).  In  the  individual  differences 
strategy,  all  participants  interact  with  the  same  microworld  under  the  same  conditions.  Analyses  are 
conducted  to  explain  differences  in  participants’  scores  in  terms  of  performance  on  psychometric 
tests  (e.g.,  intelligence  test)  or  in  terms  of  their  behaviour  while  performing  the  task.  The  aim  of  the 
individual  differences  method  is  to  find  a  way  to  predict  task  performance  and  to  gain  a  better 
understanding  of  the  demands  that  microworlds  place  on  decision  makers.  In  the  experimental 
strategy,  properties  of  the  microworld  are  varied  and  the  variations  are  assessed  using  between-  or 
within-subjects  designs.  This  method  provides  information  as  to  what  people  can  and  cannot  do 
under  differing  circumstances. 

Results  from  research  using  the  individual  differences  strategies  suggest  that  people  can  be  trained 
to  cope  with  complex  dynamic  tasks  (Brehmer,  2000).  Microworld  research  indicates  that 
individuals  who  perform  well  on  complex  dynamic  tasks  behave  in  a  systematic  way,  collect  more 
information  before  making  decisions,  evaluate  their  decisions  more  carefully,  test  causal 
hypotheses  about  the  system,  and  exhibit  more  self-reflection  than  those  who  do  not  do  well. 
Schaub  and  Strohschneider  (1989;  as  cited  in  Brehmer,  2000)  found  that  corporate  managers 
showed  the  same  kinds  of  behaviours  as  college  students,  but  with  greater  frequencies  of  the 
beneficial  behaviours.  However,  the  particular  factors  leading  to  improved  performance  are  not 
known.  Repeated  exposure,  instruction  in  systems  thinking,  and  instruction  in  the  behaviours 
needed  to  perform  better  in  complex  microworlds  do  not  seem  to  lead  to  better  performance, 
whereas  systematic  reflection  and  learning  specific  rules  for  handling  complexity  seem  to  help 
performance  (Brehmer,  2000). 

Microworlds  have  also  been  used  to  measure  key  aspects  of  DDM.  As  noted  earlier,  Fu  and 
Gonzalez  (2006)  used  the  beer  game  microworld  to  understand  how  people  learn  to  identify 
relevant  or  irrelevant  information  in  DDM.  Gonzalez  (2005)  also  used  microworlds  to  understand 
how  feedback  is  managed  in  DDM.  According  to  Gonzalez  (2005),  there  are  three  types  of  support 
of  DDM:  outcome  feedback,  cognitive  feedback,  and  feedforward  loops.  Dynamically  complex 
environments  involve  time  delays  and  decisions  that  influence  each  other  (positively  or  negatively) 
over  time.  In  her  study,  Gonzalez  used  a  microworld  to  simulate  a  Water  Purification  Plant  in  order 
to  compare  forms  of  feedback.  Participants  were  tasked  to  process  the  maximum  amount  of  water 
through  a  purification  plant  by  either  plugging,  or  emptying  water  basins.  This  is  a  highly  time 
sensitive  task  with  a  specific  window  of  time  for  each  basin  to  fill  and  be  emptied.  Successful 
control  in  this  microworld  environment  depends  on  precise  timing.  Systemic  delays  and  deadlines 
are  incorporated  that  adds  complexity  to  the  system. 

Gonzalez  (2005)  assigned  88  participants  into  5  conditions:  one  control  group  one  feedback  group, 
and  three  feedforward  groups.  In  the  control  condition,  participants  were  provided  with  the  total 
number  of  gallons  remaining  in  the  system  after  expiration  of  all  deadlines.  In  the  feedback 
condition,  participants  were  provided  with  performance-outcome  feedback  which  consisted  of  the 
number  of  gallons  of  water  remaining  in  the  system  for  each  deadline  on  the  screen.  In  the  first 
feedforward  condition  (Self-Exemplar),  participants  had  the  opportunity  to  reply  their  previous 
trials.  In  the  second  feedforward  condition  (Feedback-Exemplar),  participants  were  provided  with 
detailed  feedback  and  had  the  opportunity  to  reply  their  own  previous  trials.  In  the  final 
feedforward  condition  (Expert-Exemplar),  participants  had  the  opportunity  to  reply  the  previous 
trails  of  a  highly  skilled  participant  rather  than  their  own  previous  trials. 
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The  results  of  this  study  suggest  that  outcome  feedback  alone  is  not  an  effective  form  of  decision 
support.  That  is,  more  detailed  and  frequent  outcome  feedback  did  not  help  improve  performance 
on  DDM  tasks.  In  fact,  the  only  decision  aid  that  was  successful  was  being  able  to  see  the  previous 
trials  of  a  highly  skilled  participant  or  “expert”. 

Microworlds  have  also  been  used  to  understand  DDM  in  a  military  context.  For  example,  Bakken 
and  Gilljam  (2003)  used  a  microworld  simulation  to  conduct  research  on  Combat  Dynamic 
Intuition  (CDI).  CDI  refers  to  a  military  commander’s  ability  to  think  and  act  in  complex  and 
highly  uncertain  situations  (Bakken,  1993;  as  cited  in  Bakken  &  Gilljam,  2003).  In  particular, 
Bakken  and  Gilljam  were  interested  in  using  Minimalist  Decision  Training  to  test  CDI. 
Minimalistic  Decision  Training  (MDT)  was  designed  to  be  a  simple  and  pedagogically  focused 
simulation-supported  system  for  the  use  of  training  of  General  level  commanders.  MDT  scenarios 
put  a  commander  or  command  group  in  charge  of  his  own  logistics  and  operations  resources,  but 
simplifies  the  operating  environment  by  compressing  time  and  space. 

To  test  a  prototypical  MDT  microworld,  Bakken  and  Gilljam  (2003)  had  84  participants  from  local 
military  academies  and  a  business  school  take  part  in  a  study  that  simulated  a  humanitarian 
situation  in  which  vaccines  had  to  be  transported.  Participants  were  required  to  match  supplies  with 
capacity,  order  ahead  (while  taking  into  account  transportation  lags  and  supply  chains),  cut-off  in 
ordering  (when  capacity  utilization  approached  maximum),  and  balance  the  two  supply  chains. 

The  complexity  of  the  microworld  was  varied  through  different  levels  of  command.  Participants 
were  in  command  of  an  operational  supply  chain,  a  supporting  supply  chain,  or  both  in 
combination.  In  addition,  participants  either  read  a  brief  or  detailed  cover  story.  Finally,  prior  to 
starting  the  trials,  participants  either  practiced  on  each  of  the  single  supply  lines  in  turn  before 
embarking  on  the  combined  task  (Simplified  Approach)  or  practice  on  the  combined  task  from  the 
start  (Full-Scale  Approach). 

The  researchers  found  that  participants  who  read  the  simple  cover  story  performed  better 
(M=105.5)  than  participants  who  read  the  more  elaborate  cover  story  (M=94.5),  although  this 
difference  was  not  significant  (p  =  .15).  The  researchers  also  compared  the  top  25%  of  performers 
to  the  bottom  25%  of  performers  and  found  the  best  performers  completed  significantly  more 
practice  trials.  As  a  result  of  this  microworld  study,  Bakken  and  Gilljam  (2003)  suggest  that 
minimalist  decision  environments  are  required,  but  not  sufficient,  conditions  for  acquiring 
operational  skills. 

Although,  microworlds  have  been  used  as  training  environments  for  DDM,  little  is  known  about 
the  factors  that  lead  to  effective  microworld-based  training.  One  attempt  to  address  this  issue  was 
conducted  by  Jarmasz  (2006).  Jarmasz  (2006)  was  interested  in  understanding  whether  the 
accelerated  time  environments  found  in  microworlds  enhance  or  inhibit  the  learning  and  transfer  of 
complex  system  dynamics.  To  study  this,  Jarmasz  (2006)  conducted  a  pilot  study  to  examine 
whether  or  not  participants  were  able  to  learn  simple  DDM  in  an  accelerated  microworld 
environment  and  then  perform  the  same  task  in  a  similar,  slower  (or  real  time)  environment.  The 
two  main  hypotheses  motivating  the  study  were  1)  that  DDM  tasks  rely  heavily  on  pattern 
recognition  process  (Gonzalez  2003;  as  cited  in  Jarmasz,  2006)  and  2)  to  see  if  what  is  learned  in 
an  accelerated  environment  is  transferable  to  slower  (or  more  real-time)  environments.  Jarmasz’ s 
(2006)  study  used  a  micro  world  peace  support  operation  with  two  groups:  the  first  group  was 
trained  in  real-time  and  the  second  group  was  trained  in  faster-than-real-time.  Results  from  this 
research  showed  that  overall  the  DDM  task  was  difficult,  but  that  the  accelerated  training  group 
performed  equal  to  or  better  than  the  real-time  training  group.  Further,  it  was  found  that  the 
accelerated  group  learned  decision-making  strategies  that  the  slower  group  did  not. 
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Results  from  this  study  suggest  compressed-time  microworlds  can  support  training  and  transfer  of 
DDM  skills  to  real-time.  This  research  suggests  that  dynamic  pattern  recognition  can  transfer 
across  timescales  (Jarmasz,  2006).  These  results  were  confirmed  in  in  a  full-scale  follow-on  study 
which  validates  microworlds  as  a  potential  source  of  inexpensive,  flexible,  faster  training  for 
command  decision-making  (Jarmasz,  2007). 

Overall,  research  using  microworlds  suggests  that  people  can  be  trained  to  conduct  DDM  and  that 
microworlds  are  an  appropriate  platform  for  such  training.  Microworlds  have  also  been  shown  to 
be  useful  in  understanding  DDM  in  military  contexts  as  training  using  full-scale  exercises  or 
mission  experience  can  be  time  consuming,  expensive  and  dangerous  (Jarmasz,  2006).  However, 
concerns  have  been  raised  about  using  micro  worlds  to  investigate  DDM  (Gonzalez,  2005).  In 
particular,  concerns  have  been  raised  about  the  high  variability  in  microworld  characteristics.  That 
is,  although  microworlds  are  often  complex  and  opaque,  they  vary  considerably  in  terms  of 
dynamics  and  dynamic  complexity.  Furthermore,  researchers  have  less  experimental  control  when 
using  a  microworld  than  they  do  in  laboratory  settings.  Addressing  these  two  concerns  will 
improve  the  effectiveness  of  microworlds  to  measure  DDM. 

3.4  Summary  of  Dynamic  Decision-Making 

As  a  whole,  then,  dynamic  decision-making  is  required  in  environments  with  high  risk  and 
complexity,  and  involves  the  performance  of  tasks  requiring  multiple  steps,  that  are  inherently  time 
sensitive,  interdependent,  and  which  exert  influence  over  the  surrounding  environment  as  well  as 
being  influenced  by  it.  Decision  makers  are  required  to  perform  the  core  task  at  hand,  while 
simultaneously  having  to  make  allowances  for  the  secondary  effects  of  their  efforts.  This  is  often 
challenging  because  DDM  environments  can  involve  coordination  among  multiple  players,  and  the 
need  to  maintain  awareness  both  of  what  these  players  are  doing,  and  of  the  problem  space  as  a 
whole. 

Dynamic  decision-making  has  been  explored  from  different  perspectives,  including  systems  theory, 
psychology,  and  control  theory  from  the  engineering  domain.  These  perspectives  put  varying 
amounts  of  focus  on  different  aspects  of  DDM.  From  the  systems  theory  perspective,  for  example, 
the  DDM  environment  is  a  dynamic  system  that  ebbs  and  flows,  and  which  is  governed  by  the 
varying  forms  of  feedback  received.  The  focus  in  this  perspective  is  on  creating  analogies  that 
attempt  to  simulate  the  processes  that  people  use  to  manage  complex  systems.  These  analogies, 
moreover,  represent  one  way  to  help  people  to  form  more  accurate  models  of  the  system.  From  the 
psychological  perspective,  on  the  other  hand,  the  emphasis  is  often  less  holistic,  and  focuses  on  the 
sets  of  choices  that  people  make  as  they  attempt  to  make  complex  decisions.  Much  of  the  emphasis 
in  more  recent  years  has  been  on  the  intuitive  processes  that  people  use  to  make  complex  decisions 
under  pressure.  Decision-making,  learning,  and  planning  are  all  key  focal  points  of  the 
psychological  approach  to  understanding  DDM.  Control  theory  has  been  promoted  as  another 
theoretical  framework  that  could  be  used  to  understand  DDM.  This  theory  seems  to  have  most 
closely  aligned  with  systems  theory  (e.g.,  both  emphasize  the  importance  of  feedback),  but  seems 
to  focus  on  systems  that  are  more  constrained  and  simplistic  than  typically  the  case  within  systems 
theory.  Control  theory’s  primary  emphasis  is  on  the  role  of  feedback  while  managing  a  complex 
system. 

Much  of  the  research  exploring  dynamic  decision-making  has  used  micro  worlds  to  explore 
different  aspects  of  DDM.  This  research  has  often  explored  the  problems  that  people  have 
understanding  system  dynamics.  For  example,  some  researchers  have  argued  that  people  tend  to 
have  a  poor  understanding  of  the  building  blocks  of  system  dynamics,  including  stock  and  flows 
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and  time  delays  (Cronin  &  Gonzalez,  2007).  Research  from  the  psychological  perspective,  on  the 
other  hand,  often  has  difficulties  with  DDM  in  terms  of  decision-making  inadequacies  (e.g., 
overoptimism,  overreliance  on  heuristics).  What  seems  common  to  all  of  these  approaches, 
however,  is  that  each  makes  assumptions  that  whether  forming  models  of  complex  systems  or 
making  intuitive  decisions  based  on  very  little  information,  people  tend  to  form  some  sort  of 
mental  model  to  undertake  DDM.  These  models  are  posited  to  help  simplify  the  environment,  and 
to  make  DDM  easier  and  more  manageable.  The  exact  nature  of  these  models,  however,  is  the  topic 
of  the  next  section  on  mental  models. 
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This  chapter  explores  the  concept  of  mental  models.  The  first  section  explores  general  definitions 
of  the  concept,  followed  by  discussion  of  the  different  types  or  approaches  to  understanding  mental 
models  noted  in  the  literature,  and  comparison  of  these  approaches.  Subsequent  sections  discuss 
how  mental  models  are  formed  and  applied,  and  both  strengths  and  limitations  of  the  mental  model 
concept. 

4.1  Definitions  of  Mental  Models 

Mental  models  research  is  prolific  and  has  been  used  in  many  fields  of  study.  The  notion  of  mental 
models  has  been  used  in  domains  as  distinct  as  cognitive  psychology,  business  systems  dynamics 
research,  and  human  computer  interaction.  Specific  studies  of  cognition  and  learning,  including 
sentence  comprehension,  research  on  deductive  reasoning,  and  causal  reasoning  have  provided 
more  detailed  information  on  mental  models  (Klein  &  Hoffman,  2008). 

At  a  broad  level,  mental  models  can  be  described  as  personal  mental  representations  of  our  world. 
In  fact,  Toffler  (1970;  as  cited  in  Wilson  &  Rutherford,  1989)  defined  mental  models  as  subjective 
representations  of  external  reality  that  everyone  carries  within  their  head.  In  reviewing  the  mental 
model  literature,  Wilson  and  Rutherford  (1989)  found  consensus  from  literature  that  operators  are 
assumed  to  have,  or  to  form,  an  internal  picture  of  system  processes.  These  representations  form 
the  basis  of  mental  models.  They  went  on  to  define  such  “pictures”  or  representations  as  being 
concrete  or  abstract,  dynamic  or  static. 

Mental  models  are  generally  recognized  to  serve  three  key  functions:  to  describe,  to  predict,  and  to 
explain  our  world.  For  instance,  Markman  and  Gentner  (2001)  defined  mental  models  as 
representations  “of  some  domain  or  situation  that  supports  understanding,  reasoning,  and 
prediction”  (p.  228).  Klein  and  Hoffman  (2008)  argue  that  mental  models  help  people  to  generate 
expectancies  and  to  plan  future  courses  of  action.  Somewhat  more  specifically,  Kolkman,  Kok,  and 
van  der  Veen  (2005)  describe  mental  models  as  residing  “in  the  mind  of  individual  persons,  and 
determin[ing]  what  data  an  actor  perceives  in  the  real  world,  and  what  knowledge  the  actor  derives 
from  it.  The  mental  model  acts  as  the  filter  through  which  the  actor  observes  the  problem  situation” 
(p.  320).  In  this  sense,  mental  models  help  us  to  structure  our  experience,  as  well  as  help  to  filter 
the  information  necessary  to  understand  our  environment. 

Rouse  and  Morris  (1986)  describe  mental  models  as  “the  mechanisms  whereby  humans  are  able  to 
generate  descriptions  of  system  purpose  and  form,  explanations  of  system  functioning  and 
observed  system  states,  and  predictions  of  future  system  states.”  (p.  351).  A  pictorial  representation 
of  this  definition  can  be  found  in  Figure  1. 
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Figure  1:  Purposes  of  mental  models  (Rouse  &  Morris,  1986) 

As  can  be  seen  in  the  left  column  of  Figure  1 ,  the  main  purposes  of  mental  models  are  to  describe, 
explain,  and  predict  a  system.  The  ability  to  describe,  explain  and  predict  a  system  are  directly 
related  to  understanding  the  purpose  (i.e.,  why  a  system  exists),  function  (i.e.,  how  a  system 
operates),  state  (i.e.,  what  a  system  is  doing)  and  form  (i.e.,  what  a  system  looks  like)  of  the 
system.  Specifically,  the  ability  to  describe  a  system  allows  us  to  understand  the  purpose  of  the 
system  as  well  as  the  form  of  a  system.  The  ability  to  explain  the  system  allows  us  to  understand 
the  function  of  the  system  as  well  as  the  state  of  the  system.  Finally,  the  ability  to  make  predictions 
about  a  system  also  allows  us  to  understand  the  state  of  the  system. 

The  current  review  aims  to  explore  how  mental  models  are  understood  across  a  range  of  relevant 
domains  in  which  the  concept  is  used.  It  should  be  noted  here  that  our  objective  for  this  review  is 
not  to  identify  one  clear  and  definitive  definition  of  mental  models,  but  rather  to  describe  the 
existing  definitions  and  discrepancies  within  the  mental  models  literature.  We  will,  therefore,  spend 
the  remainder  of  this  section  discussing  several  types  of  mental  models,  including  propositional 
models,  situation  models,  system  dynamic  models,  and  physical  system  models. 
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4.2  Approaches  to  Understanding  Mental  Models 

Mental  models  research  has  been  prolific  in  many  areas  of  study.  In  this  section  we  will  discuss 
propositional  models,  mental  models  of  physical  systems,  situation  models,  and  mental  models  in 
system  dynamics'. 

In  developing  a  taxonomy  of  mental  models,  Moray  (1996)  categorized  different  approaches  to 
mental  models  in  terms  of  how  these  approaches  treated  the  relationships  between  humans,  the 
tasks  they  perform  and  their  environments.  Moray  included  figures  similar  to  the  one  below 
(containing  human,  task  and  environment  components),  but  with  added  interactivity  arrows 
between  the  three  parts.  Where  applicable,  examples  from  this  taxonomy  are  included  in  the 
sections  below. 


Figure  2:  Components  of  Moray’s  (1996)  taxonomy 


4.2.1  Propositional  Models 

One  of  the  biggest  names  in  mental  models  research  is  Johnson-Laird.  Johnson-Laird’s  approach 
to  mental  models  focuses  on  the  role  of  building  mental  models  in  order  to  perform  deductive 
reasoning.  Commonly  called  Propositional  Models,  these  types  of  mental  models  have  been  studied 
extensively  by  others  beyond  as  well.  The  primary  focus  of  this  approach  emphasizes  the  role  of 
mental  models  in  syllogistic  inferences.  To  some,  this  approach  limits  the  extent  of  mental  models, 
and  is  not  accepted  by  all  mental  models  researches  as  being  the  most  applicable  perspective 
(Staggers  &  Norcio,  1993).  The  process  of  understanding  a  syllogism  requires  “arguments  from 
premises  to  an  inference  or  conclusion”  (Stylianides  &  Stylianides,  2007,  p.  3).  To  do  this, 
Johnson-Laird  argues,  people  first  construct  a  model  of  the  information  contained  in  the  initial 
premises  of  the  syllogism.  He  argues  that  reasoning  depends  on  imagining  the  possibilities 
compatible  with  the  premises,  and  drawing  conclusions  from  these  mental  representations.  The 
representations  are  created  based  on  a  series  of  propositions  that  include  quantitative  descriptors 
(e.g.  all,  none,  some)  of  the  general  relationships  (e.g.  if,  not,  or,  and)  between  a  limited  number  of 
sets.  For  example,  if  faced  with  the  premise  of  “Some  of  the  chefs  are  musicians”.  None  of  the 
musicians  is  a  painter”  (Johnson-Laird,  2006,  p.  219),,  the  first  step  would  be  to  form  an  image  of  a 


*  This  section  will  not  include  a  discussion  of  mental  models  in  control  theory  as  no  relevant  literature  in 
control  theory  was  found  relating  to  mental  models.  This  is  not  surprising,  as  control  theory  is  generally 
focused  on  an  engineering  perspective.  That  said,  the  importance  of  feedback  in  mental  models  (which  is 
central  to  control  theory)  was  found,  and  was  included  in  the  ‘System  Dynamics’  section.  Further,  how  users 
create  mental  models  of  physical  systems  was  also  found  and  is  included  in  the  ‘Physical  Systems’  section. 
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group  of  people.  This  mental  model  has  a  structure  that  is  similar  to  that  of  the  situation  it 
represents.  At  the  next  stage  of  reasoning,  some  of  the  chefs  would  be  holding  musical  instruments, 
and  others  would  be  holding  no  musical  instruments.  Another  set  of  people  would  be  holding 
paintbrushes.  Thus  as  more  data  is  provided  (or  perceived)  in  a  situation,  that  new  piece  of 
information  is  synthesized  into  the  model,  so  that  with  each  additional  bit  of  information,  the  model 
is  then  adjusted.  If  more  than  one  model  is  seen  to  be  consistent  with  the  premises,  other  models 
might  be  constructed.  The  decision  maker  then  searches  for  evidence  in  support  of  the  model,  and 
makes  some  sort  of  conclusion.  Then,  the  decision  maker  will  search  for  “alternative  mental 
models  that  may  lead  to  refutation  of  the  conclusion  (counterexamples)”  (Stylianides  & 

Stylianides,  2007,  p.  3). 

The  ability  to  form  mental  models,  Stylianides  and  Stylianides  (2007)  argue,  is  dependent  on 
several  core  abilities.  Key  amongst  these  is  the  ability  to  understand  language,  the  ability  to  apply 
rules  of  logic,  and  the  ability  to  search  for  alternative  models.  These  skills  develop  as  people 
mature,  in  conjunction  with  the  increasing  capacity  to  use  working  memory.  In  fact,  they  argue  that 
errors  in  deductive  reasoning  often  occur  simply  because  people  fail  to  consider  all  the  possible 
counterexamples  to  the  mental  model  that  they  create. 

Through  numerous  reasoning  studies  Johnson-Laird  (2001,  2005)  has  made  a  number  of 
conclusions  and  predictions  about  mental  models,  as  follows: 

Mental  models  are  iconic.  Accordingly,  the  mental  model  takes  the  same  structure  as  the 
object  the  model  is  a  representation  of  as  far  as  possible,  but  certain  components  of  them 
are  necessarily  symbolic.  Johnson-Laird  emphasizes  that  this  iconicity  is  important. 

One  model  is  better  than  many  models  in  terms  of  improving  reasoners’  abilities.  The 
fewer  the  number  of  mental  models  needed  for  an  inference,  and  the  simpler  they  are,  the 
easier  inference  should  be.  It  should  take  less  time,  and  be  less  prone  to  error.  Otherwise,  if 
multiple  models  are  required,  reasoners  sometimes  fail  to  consider  all  models.  This 
prediction  is  a  consequence  of  the  limitations  of  working  memory  (Johnson-Laird,  2001). 

Mental  models  represent  what  is  true,  but  not  what  is  false.  Any  manipulation  that  helps 
reasoners  bring  to  mind  the  falsifying  instance  of  the  conditional  improves  performance  in 
the  selection  task  (Johnson-Laird,  2005).  So  reasoners  can  -  with  some  difficulty  -  flesh 
the  models  out  into  fully  explicit  models.  However,  when  falsity  matters,  fallacies  do 
occur. 

Formation.  Content  and  background  knowledge  modulate  interpretations  of  the  world  so 
mental  models  are  created  to  be  consistent  with  the  premises  understood  by  the  reasoner. 
This  means  that  models  are  not  necessarily  based  on  unbiased  truths,  and  instead  do,  at 
times,  include  inaccuracies. 

Research  led  by  Johnson-Laird  demonstrates  that  the  quality  and  availability  of  mental  models  has 
an  important  impact  on  human  reasoning  ability  (Sparkes  &  Huf,  2003).  Through  proper  use  of 
mental  models  our  reasoning  ability  is  able  to  improve,  and  with  experience  reasoners  are  able  to 
develop  tailor-made  strategies  for  particular  sorts  of  problems. 

Moray’s  (1996)  representation  for  the  abstract  problem  solving/propositional  logic  approach 
(typical  of  Johnson-Laird)  is  shown  in  Figure  3. 
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Figure  3.  Mental  models  for  understanding  propositional  logic  (Moray,  1996) 

As  the  task  is  typically  an  abstract  reasoning  problem  provided  to  participants  (e.g.,  patterns  on 
cards,  verbal  descriptions),  Moray  (1996)  reasoned  that  the  only  relevant  link  is  from  the  task  to  the 
human.  The  information  about  the  reasoning  task  becomes  a  mental  model  because  the  information 
is  turned  into  a  mental  representation,  which  is  manipulated  by  the  participant  to  generate  an 
answer  (hence  the  feedback  loop  on  the  human  dimension).  Importantly,  with  this  approach,  then, 
the  human  is  influenced  by  the  task,  but  does  not  influence  it.  Interactions  with  the  environment  are 
also  not  prominent  in  this  conceptualization  of  mental  models. 

There  are  a  number  of  factors  limiting  the  application  of  propositional  models  to  dynamic  decision¬ 
making.  Mainly,  this  approach  seems  to  focus  on  a  very  small  subset  of  reasoning  tasks  that 
involve  static  finite  constructs.  Further,  there  is  no  interaction  between  effects  of  decision  makers 
on  the  task  or  on  the  environment.  Thus,  these  scenarios  share  little  similarity  to  the  complex, 
interdependent,  and  uncertain  environments  of  dynamic  decision-making.  Still,  there  are  a  number 
of  considerations  about  mental  models  that  will  be  taken  forward  from  this  perspective.  Namely, 
that  humans  have  problems  dealing  with  more  than  one  mental  model,  that  mental  models  do  not 
tend  to  contain  what  is  false  and  that  mental  models  are  formed  based  on  the  reasoners’  perspective 
and  are  not  always  accurate. 

4.2.2  Physical  Systems 

Mental  models  have  also  been  used  to  describe,  explain  and  predict  the  workings  of  physical 
systems.  Forbus  and  Gentner  ( 1 997)  define  mental  models  as  the  models  people  use  in  reasoning 
about  the  physical  world.  Mental  models  are  important  in  reasoning  about  complete  physical 
systems,  in  making  and  articulating  predictions  about  the  world,  and  in  discovering  causal 
explanations  for  what  happens  around  us.  In  her  discussion  of  mental  models,  Gentner  (2002; 
Markham  &  Gentner,  2001)  distinguishes  between  the  logical  and  the  causal  mental  models 
approach.  The  logical  models  approach  focuses  on  mental  models  as  working  memory  constructs 
that  support  logical  reasoning  (e.g.,  Johnson-Laird).  However,  Gentner’ s  focus  is  on  the  causal 
models  approach.  This  approach  seeks  to  characterize  the  knowledge  and  processes  that  support 
understanding  and  reasoning  about  physical  systems  and  mechanisms.  Markman  and  Gentner 
(2001)  define  causal  mental  models  as  “mental  representations  that  are  used  in  reasoning  and  that 
are  based  on  long-term  domain  knowledge  or  theories”  (p.  229).  In  particular,  causal  mental 
models  are  used  to  explain  reasoning  about  physical  systems  and  mechanisms  (e.g.,  human- 
computer  interaction,  ecology). 
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Figure  4.  Mental  models  for  understanding  physical  systems  (Moray,  1996) 

With  respect  to  physical  devices,  Moray  (1996)  states  that  mental  models  are  developed  as 
information  predominantly  flows  from  the  device  to  the  person.  The  information  received  then 
becomes  incorporated  into  the  mental  model.  The  contents  of  the  mental  model  are  examined  and 
manipulated  to  construct  a  causal  account  of  the  functioning  of  the  device.  The  environment  does 
not  figure  prominently  in  this  process.  For  human-computer  interactions,  the  mental  model  is  seen 
as  a  generator  of  action.  Therefore,  the  emphasis  is  on  action  generation,  represented  by  an  arrow 
from  the  operator  to  the  task.  Within  large-scale  industrial  systems,  the  systems  themselves  are  too 
large  for  operators  to  be  able  to  track  exact  values  of  all  variables  from  moment  to  moment.  As  the 
sheer  size  of  large  industrial  systems  does  not  allow  operators  to  be  frilly  up  to  date  on  the  state  of 
the  system,  operators  depend  upon  their  mental  models  for  sufficient  understanding  to  control  the 
system. 

In  addition  to  physical  systems  and  mechanisms,  mental  models  can  be  used  to  reason  about 
physical  events  (Markham  &  Gentner,  2001).  Such  models  incorporate  a  small  number  of  variables 
about  an  event,  which  variables  are  often  qualitative  rather  than  quantitative.  That  is,  people  reason 
about  relative  properties  of  physical  systems  (e.g.,  direction  of  motion,  relative  speed,  relative 
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mass)  but  do  not  estimate  values  in  specific  quantities  or  carry  out  mathematical  simulations  of  a 
system’s  behaviour(Markman  &  Gentner,  2001).  Over  time,  people  typically  possess  multiple 
models  of  complex  systems,  some  highly  context-bound  and  others  more  abstract.  This  is 
consistent  with  work  done  by  Rasmussen,  Pejtersen,  &  Schmidt  (1990)  that  looks  at  explaining  the 
same  system  at  variable  levels  of  abstraction  when  designing  interfaces  for  complex  systems. 
Rasmussen  (1979)  states  that  “a  system  can  be  viewed  as  what  it  looks  like,  how  it  functions  or 
why  it  exists.  All  of  these  views  are  ‘correct’  and  of  value  for  answering  a  variety  of  questions 
about  a  system”  (as  cited  in  Rouse  &  Morris,  1986,  p  353). 

According  to  the  physical  systems  domain,  operators  form  a  representation  of  the  physical  system 
in  their  working  memory  (WM)  by  combining  knowledge  stored  in  their  long-term  memory  (LTM) 
and  using  the  information  they  acquire  about  task  characteristics  (Genter  &  Stevens,  1983;  as  cited 
in  Canas,  Antoli,  &  Quesada,  2001).  When  dealing  with  physical  systems,  efforts  are  often  focused 
on  how  information  about  the  system  representation  is  acquired  and  extracted  from  LTM  (Canas  et 
al.,  2001). 

Mental  models  can  serve  a  number  of  functions,  such  as  enabling  mental  simulation  and  promoting 
learning  processes.  Simulations  enable  running  a  mental  model  internally,  in  order  to  observe  how 
it  will  behave  and  what  the  outcome  will  be.  Mental  simulations  often  involve  the  use  of  imagery 
(Markham  &  Gentner,  2001).  Furthermore,  mental  models  can  facilitate  learning,  particularly  when 
the  structure  of  the  new  learning  is  consistent  with  the  model.  For  example,  Kieras  and  Bovair 
(1984;  as  cited  in  Gentner,  2002)  showed  that  participants  could  operate  a  simulated  device  more 
accurately  and  could  diagnose  malfunctions  better  when  they  had  a  causal  mental  model  of  its 
functioning  rather  than  a  mere  procedural  grasp  of  how  the  device  operated. 

According  to  Gentner  (2002)  people  can  hold  two  or  more  inconsistent  mental  models  within  the 
same  domain.  For  example,  a  novice  learner  may  give  one  explanation  of  why  a  towel  dries  in  the 
sun  and  another  explanation  of  what  causes  a  puddle  of  water  to  evaporate  without  seeing  any 
connection  between  the  two  situations  (Collins  &  Gentner,  1987;  as  cited  in  Gentner,  2002).  In 
fact,  Gentner  (2002)  states  that  “novices  often  use  locally  coherent  but  globally  inconsistent 
accounts,  often  quite  closely  tied  to  the  details  of  the  particular  example.  This  pattern  emphasizes 
the  tendency  of  novices  to  learn  conservatively,  with  knowledge  cached  in  highly  specific,  context- 
bound  categories”  (p.  9685).  However,  people  shift  to  using  learned  rules  rather  than  mental 
models  for  reasoning  as  they  gain  experience  with  a  domain  (Markham  &  Gentner,  2001).  For 
example,  Schwartz  &  Black  (1996;  as  cited  in  Markham  &  Gentner,  2001)  asked  participants  to 
solve  a  number  of  gear  problems.  One  such  problem  stated  that  there  are  7  gears  on  a  pegboard  are 
arranged  in  a  circle  so  that  each  gear  meshes  with  the  one  next  to  it.  If  one  gear  was  rotated 
clockwise,  which  way  would  the  next  gear  move?  When  first  asked  the  question,  participants 
mentally  simulated  the  motion  of  the  gears,  often  with  accompanying  hand  gestures.  Over  time, 
though,  people  learned  the  parity  rule  (that  every  gear  in  a  sequence  turns  in  the  same  direction,  but 
adjacent  gears  turn  in  the  opposite  direction),  and  shifted  from  using  their  mental  models  to  rules  to 
solve  the  problems.  Therefore,  it  appears  that  with  experience  people  access  their  stored  knowledge 
of  the  outcome  to  search  for  rules  to  help  solve  problems  rather  than  carry  out  full  mental 
simulations  of  the  system’s  behaviour. 

The  literature  on  physical  systems  raises  a  number  of  issues  pertaining  to  the  ability  of  mental 
models  to  support  DDM.  For  example,  Gentner’s  (2002)  definition  of  causal  mental  models  (i.e., 
characterize  the  knowledge  and  processes  that  support  understanding  and  reasoning  in  knowledge 
rich  domains)  suggest  that  such  models  would  be  useful  in  supporting  DDM.  However,  there  was 
little  mention  in  the  physical  systems  literature  reviewed  of  how  such  mental  models  are  created  or 
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work  in  DDM  environments.  There  is,  however,  an  underlying  assumption  is  that  it  is  possible  to 
have  a  coherent  mental  model  of  the  physical  system,  i.e.  that  the  system  is  not  too  complex  for  the 
mind.  Whether  this  is  accomplished  through  having  one  mental  model  or  multiple  models  of  the 
same  system  at  different  levels  of  abstraction  would  be  important  to  know.  In  addition,  the  role  of 
memory  types  (i.e.  working  memory  versus  long  term  memory)  in  creating  and  maintaining  a 
mental  model  needs  further  discussion  to  understand  how  they  would  impact  a  decision  maker’s 
ability  to  conduct  DDM. 

4.2.3  Situation  Models 

4.2.3.1  Situation  modeis  as  situation  awareness 

Situation  models  seem  to  be  a  more  constrained  form  of  mental  models.  The  situation  model  is  “the 
current  instantiation  of  the  mental  model  which  is  more  general  in  nature”  (Endsley,  2000a,  p.  2). 
Endsley  (1995)  defines  a  situation  model  as  “a  schema  depicting  the  current  state  of  the  mental 
model  of  the  system”  (p.  43).  It  is  a  dynamic  representation  of  a  person’s  knowledge  and 
understanding  of  the  present  state  of  a  system  (Endsley,  2000a).  A  situation  model  may  incorporate 
the  value  of  different  system  parameters  (e.g.,  level  of  fuel  gauge),  an  understanding  of  the 
dynamics  of  the  system  (e.g.,  rate  of  change)  developed  from  changes  in  the  situation  model  over 
time,  higher  level  comprehension  (e.g.,  understanding  the  state  of  the  system  in  a  gestalt  form  in 
relation  to  human  goals),  and  projections  of  future  states.  A  situation  model  may  also  assign  well 
understood  situations  to  prototypical  classifications  of  the  system  state. 

In  understanding  this  definition  of  a  situation  model,  it  is  crucial  to  know  that  Endsley  (2000a)  uses  the 
two  terms  “situation  models”  and  “situation  awareness”  (SA)  synonymously.  She  defines  situation 
awareness  as  “a  person’s  mental  model  of  the  world  around  them”  or  a  situation  model  (p.  2). 
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Figure  5:  Situation  Model  in  dynamic  decision-making  (Endsley,  2000b) 

Figure  5  is  a  model  of  the  overall  decision-making  process  and  provides  the  basis  for  understanding 
situation  models  within  decision-making  (Endsley,  1995).  According  to  this  model,  people’s 
perceptions  of  their  environments  form  the  basis  of  their  situation  model.  Action  selection 
(Decision)  and  performance  (Performance  of  Actions)  are  separate  stages  that  proceed  directly 
from  the  situation  model.  Factors  that  influence  the  decision-making  process  include  people’s 
information  processing  mechanisms,  preconceptions,  goals  and  objectives. 

Of  importance  to  our  discussion  are  the  three  levels  that  make  up  situation  models  (Endsley, 

2000b). The  first  level  is  perception  of  cues.  One  needs  basic  perception  of  important  information  in 
order  to  form  a  correct  picture  of  the  situation.  The  second  level  is  comprehension,  which  refers  to 
one’s  ability  to  integrate  multiple  pieces  of  information  and  determine  their  relevance  to  one’s 
goals.  The  final  level  is  projection,  which  refers  to  the  ability  to  project  future  events  from  current 
events  and  dynamics. 

Time  also  plays  an  important  role  in  the  formation  of  a  situation  model  (Endsley,  2000b).  A  critical 
part  of  the  situation  model  is  knowing  how  much  time  is  left  until  some  event  occurs  or  until  an 
action  must  be  taken.  “The  rate  at  which  information  is  changing  is  a  part  of  SA  regarding  the 
current  situation,  which  also  allows  for  projection  of  future  situations’’  (p.  4).  As  situations  are 
dynamic,  one’s  situation  model  must  constantly  change  or  be  inaccurate. 

Also  of  note  in  Figure  5  is  Endsley’s  (2000b)  vision  of  situation  model  as  a  state  separate  from 
decision-making  and  performance.  The  situation  model  is  depicted  as  the  operator’s  internal  model 
of  the  state  of  the  environment.  Based  on  this  representation,  operators  can  decide  what  actions  to 
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take  to  deal  with  the  situation.  Thus,  Endsley  envisions  situation  models  as  the  main  precursor  to 
decision-making.  She  argues  this  is  why  it  is  possible  to  have  a  perfect  situation  model  and  still 
make  an  incorrect  decision.  For  example,  a  battle  commander  can  know  where  an  enemy  and  what 
the  enemy  is  capable  of,  but  decide  upon  inadequate  strategies  or  tactics  to  launch  an  attack. 

Factors  that  impact  the  accuracy  and  completeness  of  situation  models  include  attention  and 
working  memory  (Endsley,  2000b).  Where  people  direct  their  attention  has  a  fundamental  impact 
on  the  aspects  of  the  environment  incorporated  into  a  situation  model.  Novice  decision  makers  and 
those  in  novel  situations  must  combine  information,  interpret  the  information,  and  strive  to  make 
projections  within  limited  working  memory. 
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Figure  6.  Relationship  between  the  mental  model  and  the  situation  model  (Endsley, 

2000a). 

With  respect  to  mental  models,  Endsley  (2000a)  states  that  mental  models  determine  what 
information  is  attended  to,  how  that  information  is  interpreted  and  integrated,  and  any  projections 
made  about  what  will  happen  to  the  system  in  the  future.  Situation  models,  on  the  other  hand,  are 
developed  by  observing  the  world  and  using  existing  mental  models.  For  example,  a  mechanic  has 
a  mental  model  of  how  an  engine  works,  as  well  as  a  situation  model  of  how  the  engine  is 
functioning  right  now  (carburetor  problems),  the  state  of  its  components  (new  or  worn),  and  its 
capabilities  and  features  (horsepower).  Such  knowledge  is  made  possible  by  perceiving, 
interpreting,  filtering  and  integrating  information.  Mogford  (1997;  as  cited  in  Endsley,  2000b) 
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states  that  the  mental  model  is  the  underlying  knowledge  base  for  a  situation  model.  The 
information  in  a  mental  model  influences  and  structures  the  situation  model  data  and  directs 
attention,  which  can  be  detrimental  when  the  selection  and  interpretation  of  information  from  the 
mental  model  results  in  a  faulty  SM  (Endsley,  2000b). 

With  respect  to  DDM,  it  appears  that  situation  models  are  highly  applicable.  Situation  models  are 
run  from  a  basic  mental  model  of  a  system,  but  relate  to  what  is  going  on  at  the  moment.  Therefore, 
they  are  always  being  updated.  In  addition,  situation  models  are  used  to  predict  what  will  happen  in 
the  system  in  the  future.  Also,  they  have  been  studied  in  complex,  uncertain  contexts  showing  that 
they  could  likely  be  used  in  DDM  environments. 

4.2.3.2  Situation  Modeis  and  Narratives 

The  basis  of  language  comprehension  is  a  reader’s  ability  to  construct  mental  models  of  the 
situation  being  described  (Bower  &  Morrow,  1990).  Comprehension  of  a  text,  thus,  requires  the 
construction  of  a  mental  representation  of  what  the  text  is  about:  a  situation  model.  With  respect  to 
narratives,  situation  models  are  “mental  representations  of  the  people,  objects,  locations,  events  and 
actions  described  in  a  text,  not  of  the  words,  phrases,  clauses,  sentences  and  paragraphs  of  a  text” 
(Zwaan,  1999,  p.  15).  The  situation  model  is  used  to  interpret  and  evaluate  later  statements  in  the 
text.  It  is  the  mental  representation,  rather  than  the  text  itself,  that  readers  remember  (Bower  & 
Morrow,  1990).  Much  of  what  a  dynamic  decision  maker  learns  about  the  world  is  through  text 
(e.g.  emails,  articles,  Powerpoint  presentations,  reports,  etc).  While  DDM  is  not  explicitly  formed 
through  narrative  text,  there  are  lessons  here  that  are  applicable. 

According  to  Bower  and  Morrow  (1990)  a  situation  model  of  a  narrative  contains  two  major  parts. 
First,  there  are  the  descriptions  of  the  cast  of  characters,  their  occupations,  relationships,  and 
personal  traits.  These  details  explain  the  characters’  goals,  plans,  and  actions  as  the  plot  develops. 
Readers  assume  that  goals  and  plans  can  explain  a  character’s  actions.  These  explanations  allow 
readers  to  build  a  network  of  causal  connections  among  events  in  the  story.  Because  goals  are  the 
most  important  causes  of  characters’  actions,  readers  set  up  a  goal  list  of  each  character  in  their 
memory  and  monitor  how  story  events  relate  to  these  goals. 

Second,  there  is  a  mental  map  of  the  physical  settings  in  which  the  actions  occur,  also  known  as  a 
spatial  model  (Bower  &  Morrow,  1990).  This  spatial  model  is  a  mental  map  of  the  places, 
landmarks,  and  objects  as  they  are  laid  out  in  the  narrative  and  the  locations  of  characters  as  they 
move  about  in  this  space.  Bower  and  Morrow  suggest  that  readers  “construct  in  imagination  a  sort 
of  theatre  stage  or  ‘doll  house’  with  landmarks  and  rooms  filled  with  expected  objects,  plus  any 
special  objects  mentioned  in  the  text”  (p.  45). 

When  we  are  engaged  in  a  narrative,  we  have  a  certain  vantage  point  (spatial,  temporal,  and 
psychological)  from  which  we  experience  the  event  vicariously.  Zwaan  (1999)  refers  to  this 
perspective  as  a  deictic  center.  The  following  aspects  of  a  narrative  are  relevant  to  a  deictic  center 
(Zwaan  &  Radvansky,  1998;  Zwaan,  1999): 

Space.  Situation  models  should  represent  relevant  aspects  of  the  environment.  Readers  are 
faster  to  recognize  words  denoting  objects  closer  to  the  protagonist  than  farther  from  the 
protagonist  (Zwaan,  1999;  Bower  &  Morrow,  1990).  When  readers  have  a  spatial  model  of 
the  narrative,  they  update  their  representations  according  to  the  location  and  goals  of  the 
protagonist  and,  subsequently,  have  the  fastest  mental  access  to  the  room  the  protagonist  is 
currently  in  or  is  heading  towards. 
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Time.  We  assume  events  are  narrated  in  chronological  order  (Zwaan,  1999).  This  is 
because  we  experience  events  in  our  lives  in  chronological  order.  Therefore,  people  who 
construct  situation  models  find  it  more  difficult  to  comprehend  text  that  is  presented  out  of 
chronological  order. 

Goals  and  causation.  Unfinished  goals  are  more  prominent  in  our  minds  than  goals  that 
have  already  been  accomplished  (Zwaan,  1999).  This  is  the  same  when  reading  text.  Goals 
that  the  protagonist  has  not  yet  accomplished  are  more  accessible  to  the  reader  than  goals 
that  have  already  been  accomplished.  We  also  have  a  strong  tendency  to  interpret  events  as 
being  causal  and,  therefore,  infer  causality. 

People  and  objects.  Readers  are  quick  to  make  inferences  about  protagonists  (e.g., 
personality  characteristics,  emotions),  presumably  in  order  to  construct  a  more  complete 
situation  model  (Zwaan,  1999).  Reading  speed  is  faster  when  stereotypes  are  held  (e.g., 
male  electrician)  than  when  they  are  violated  (e.g.,  female  electrician). 

4.2.4  System  Dynamics 

The  concept  of  mental  models  is  vitally  important  to  the  field  of  system  dynamics  (Doyle  &  Ford, 
1998).  The  information  gleaned  from  mental  models  about  the  structure  and  relationships  in 
dynamic  systems  allows  for  the  construction  of  system  dynamic  computer  models  in  the  absence  of 
written  and  numerical  data.  Consequently,  system  dynamics  researchers  have  devoted  much  effort 
in  developing  techniques  and  procedures  for  eliciting,  representing,  and  mapping  mental  models. 
Conversely,  the  goal  of  most  educational  interventions  in  system  dynamics  is  to  change  or  improve 
mental  models  in  order  to  improve  the  quality  of  dynamic  decisions. 

Unfortunately,  there  is  no  coherent  definition  of  mental  models  within  the  system  dynamics  field. 

In  order  to  provide  some  specification  and  clarity  to  the  mental  models  literature,  Doyle  and  Ford 
( 1 998)  proposed  that  system  dynamics  researchers  are  primarily  interested  in  specialized  cognitive 
structures  they  termed  “mental  models  of  dynamic  systems”  (MMODS).  The  term  “mental 
models”,  therefore,  is  an  abbreviation  of  MMODS.  MMODS  is  a  “relatively  enduring  and 
accessible,  but  limited,  internal  conceptual  representation  of  an  external  system  whose  structure 
maintains  the  perceived  structure  of  that  system”  (p.  19). 

At  the  heart  of  decision-making  within  system  dynamics  is  feedback  theory.  The  basis  for  decision¬ 
making  within  feedback  theory  is  the  cybernetic  loop  (Richardson,  Andersen,  Maxwell,  &  Stewart, 
1994).  This  loop  involves  the  state  of  the  system,  the  perceived  state,  goals,  planned  action,  and 
action  affecting  the  state  of  the  system,  and  closing  the  loop  of  perception,  planning,  and  action. 
According  to  feedback  theory  as  proposed  by  Richardson  et  al.  ( 1 994)  “a  mental  model  in  a 
dynamic,  planned  action  setting  must  be  composed  of  (at  least)  these  four  elements:  intentions, 
perceptions,  system  structures,  and  plans”  (p.  182)  as  illustrated  in  Figure  7.  This  system  dynamics 
approach,  which  Richardson  et  al.  (1994)  refer  to  as  feedback  theory,  focuses  on  the  elements  of  a 
mental  model  and  the  connections  between  those  elements. 
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Figure  7.  Components  of  mental  models  placed  in  the  context  of  the  classic 
cybernetic  process  (Richardson  et  al.,  1994). 

Within  feedback  theory,  a  mental  model  comprises  three  submodels:  the  ends  model  (goals),  the 
means  model  (strategies,  tactics),  and  the  means/ends  model  (relation  between  goals  and  strategies 
or  tactics)  (Richardson  et  al.,  1994).  The  influence  of  all  elements  of  this  view  of  a  mental  model 
(shown  in  Figure  7)  are  connected.  Flowever  “. .  .ignoring  their  distinguishable  characteristics  can 
be  misleading”  (Richardson  et  al.,  1994,  p.  183). 

•  Ends  model  focuses  on  the  goals  and  contains  perceptions  and  information  about  what  one 
is  trying  to  accomplish  in  a  decision  or  a  number  of  decisions  over  time.  There  are  local  or 
proximal  goals  (i.e.,  intermediary  goals)  and  global  or  distal  goals  (i.e.,  major  or  ultimately 
important  ends  to  strive  for).  A  person’s  ends  model  in  DDM  tasks  includes  this  set  of 
local-to-global  goals. 

•  Means  model  focuses  on  the  strategies,  tactics,  and  policy  levers  decision  makers  believe 
are  available  as  they  move  towards  their  perceived  goals. 

•  Means/ends  model  focuses  on  the  connection  between  the  ends  model  and  the  means 
model.  This  model  can  be  a  mental  representation  of  the  stock-and- flow/feedback  structure 
of  a  complex  dynamic  system.  It  could  also  be  a  simple  chain  of  associations  linking  a 
policy  lever  to  an  outcome. 

However,  Richardson  et  al.  ( 1 994)  criticize  the  feedback  theory  vision  of  mental  models  as  an 
inadequate  foundation  for  mental  model  research  on  DDM.  They  state  that  the  theory’s  treatment 
of  perception  is  woefully  weak  and  it  fails  to  incorporate  structures  for  learning.  Subsequently, 
Richardson  et  al.  proposed  their  own  vision  of  mental  models. 
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Figure  8.  Integrated  theory  of  perception,  planning,  action,  and  learning 

(Richardson  et  al.,  1994) 

In  composing  their  model,  Richardson  et  al.  (1994)  incorporated  ideas  from  the  Brunswickean  lens 
model  into  the  feedback  theory  model  to  develop  their  representation  of  mental  models.  Within  the 
Brunswickean  lens  model,  the  “perceived  state”  is  a  complex  process  involving  true  descriptors  of 
the  state  of  the  system,  cues  and  measurements  derived  from  the  descriptors,  and  subjective  cues 
(Richardson  et  al.  1994).  Subjective  cues  are  a  combination  of  objective  cues  unconsciously 
assembled  by  the  observer.  They  are  the  result  of  scanning  a  field  for  all  possible  cues,  selecting 
combinations  of  cues  to  attend  to,  and  gauging  the  cues.  These  cues  are  then  subjectively 
interpreted  in  light  of  experience,  memory,  and  perceptions.  Assessments  of  the  current  system 
state  and  predictions  for  future  states  are  made  from  these  subjective  cues. 

Within  the  Richardson  et  al.  (1994)  model,  an  observer  scans  the  system,  selects  and  interprets  cues 
from  it,  and  subjectively  interprets  them  as  a  basis  for  assessing  and  predicting  the  state  of  the 
system.  Assessments  and  predictions  are  based  on  the  observer’s  cognitive  model  of  the  system. 
Comparisons  of  the  desired  state  and  the  current  state  result  in  the  selection  of  strategies,  tactics, 
and  policy  levers.  Richardson  et  al.  also  included  mechanisms  for  learning,  as  they  felt  omitting 
this  mechanism  was  a  weakness  of  previous  models.  They  state  that  learning  is  the  mix  of 
processes  that  allow  people  to  change  their  mental  models. 

According  to  Richardson  et  al.  (1994),  their  model  incorporates  observers’  ability  to  modify  their 
thinking  to  improve  assessments,  predictions,  and/or  interventions  in  complex  dynamic  systems  by 
1)  changing  the  cues  they  pay  attention  to  and  how  they  are  interpreted,  2)  change  the  way  they 
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think  about  how  the  system  functions,  3)  change  the  strategies  and  tactics  they  are  using,  and  4) 
change  the  goals  they  strive  toward. 

Richardson  et  al.’s  (1994)  vision  of  mental  models  appears  well  suited  for  supporting  dynamic 
decision-making.  The  means/ends  model  includes  a  feedback  system  so  that  the  decision  maker  is 
able  to  make  adjustments  to  his  or  her  mental  model.  This  accounts  for  learning,  the  “. .  .process  by 
which  people  change  their  mental  models”  (Richardson  et  al.,  1994,  p.l86)  that  assist  in  creating  a 
more  accurate  mental  model  of  the  system. 

4.2.5  Overview  of  Prominent  Approaches 

The  various  approaches  to  understanding  mental  models  emphasize  different  aspects  about  them. 
In  an  attempt  to  clarify  these  different  aspects  Table  3  below  was  created.  There  was  a  great  deal 
of  discussion  about  these  mappings,  and  some  disagreement  (which  are  reflected  by  ‘?’)  due  to 
different  interpretations  within  and  across  the  approaches. 


Table  3:  Similarities  Between  Approaches 


Johnson 

Laird 

Situation 

Models 

System 

Dynamics 

Physical 

Systems 

Cognitive  representations 

V 

V 

V 

V 

Alternative  models  are  generated 

V 

V 

— 

V 

Series  of  propositions 

V 

— 

? 

— 

One  best  model 

V 

— 

— 

— 

Real-time  representation 

— 

V 

— 

— 

Content  and  background  knowledge  modulate 
interpretations 

V 

a/ 

a/ 

— 

Visual  representation 

V 

V 

? 

V 

Dynamic 

— 

< 

< 

< 

Perception  of  cues 

— 

V 

— 

— 

Includes  understanding  of  current  state 

V 

V 

V 

V 

Projection  of  future  events 

— 

V 

V 

— 

Time  critical 

— 

V 

V 

— 

Representation  includes  the 
environment/context 

— 

a/ 

a/ 

a/ 

Goal  oriented 

— 

? 

V 

V 

Used  to  explain  physical  systems  and/or  events 

— 

— 

< 

< 

There  are  very  few  similarities  between  all  of  the  mental  models  discussed  above.  The  only 
similarities  that  hold  true  are  that  mental  models  are  that  they  are  cognitive  representations  and  that 
they  include  understanding  of  the  current  state.  The  most  similar  perspectives  appear  to  be  System 
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Dynamics  and  Physical  Systems,  with  Johnson-Laird’s  propositional  models  appearing  to  be  the 
most  distinct. 

4.2.6  Challenges  to  Understanding  Mental  Models 

Our  review  suggests  two  primary  challenges  inherent  in  understanding  the  concept  of  mental 
models.  These  relate  to  the  lack  of  a  clear  definition,  and  the  related  problem  of  capturing  and 
measuring  mental  models.  Both  of  these  issues  have  been  raised  in  previous  reviews  on  mental 
models  (Rouse  and  Morris,  1986;  Staggers  &  Norcio,  1993),  though  neither  appear  any  closer  to 
being  solved. 

4.2. 6. 1  Lack  of  a  clear  definition 

Even  though  the  notion  of  mental  models  is  widely  accepted  in  many  research  fields,  there  is  no  set 
definition  of  mental  models.  The  lack  of  a  consistent  and  coherent  definition  was  indicated  in  early 
research  (e.g..  Rouse  &  Morris,  1986),  and  is  noted  even  in  more  recent  research.  Canas,  Antoli, 
and  Quesada  (2001)  state  that  “. .  .a  theoretical  problem  exists,  which  is  reflected  in  the  great 
confusion  concerning  the  definition  of  a  mental  model”.  Wilson  and  Rutherford  (1989)  argue  that 
some  definitions  of  mental  models  describe  hypothetical  constructs  that  assist  scientists  in 
understanding  behaviour,  whereas  others  are  entirely  domain-oriented.  Some  authors  attempt  to 
define  mental  models  through  comparison  with  other  structures  “Mental  models  are  best  introduced 
by  contrast  with  the  notion  of  a  schema. . .”  (Holland,  Holyoak,  Nisbett,  &  Thagard,  1986,  p  12). 
Based  on  their  literature  review,  Kolkman,  Kok,  and  van  der  Veen  (2005)  have  argued  that  the  term 
“mental  model”  is  used  in  many  disciplines,  each  having  its  own  specific  definition.  Within  distinct 
research  fields,  there  is  a  lack  of  consensus  about  what  mental  models  actually  are. 

In  the  field  of  system  dynamics,  there  has  yet  to  be  developed  an  unambiguous  and  agreed  upon 
definition  of  mental  models  (Doyle  &  Ford,  1998).  In  their  review  of  mental  models  research, 

Doyle  and  Ford  (1998)  found  the  definitions  of  mental  models  to  be  overly  brief,  general,  and 
vague.  Furthermore,  Doyle  and  Ford  found  that  authors  within  system  dynamics  and  cognitive 
sciences  disagreed  on  what  they  considered  to  be  the  basic  characteristics  of  mental  models.  In 
addition  to  a  lack  of  a  strong  definition,  there  is  also  a  lack  of  consensus  as  to  what  term  to  use  to 
define  mental  models.  For  instance,  within  the  human-computer  interaction  literature,  mental 
models  are  also  referred  to  as  conceptual  models,  cognitive  models,  component  models,  and  causal 
models  (Staggers  &  Norcio,  1993). 

The  lack  of  consensus  has  had  several  consequences  (Doyle  &  Ford,  1998).  First,  researchers  have 
developed  and  applied  idiosyncratic  conceptions  of  mental  models,  which  has  hindered 
communication  within  their  own  field  (i.e.,  system  dynamics).  In  addition,  the  various  definitions 
within  the  field  have  meant  that  different  research  techniques  are  used.  One  of  the  impacts  of  this  is 
that  research  results  cannot  be  cumulated  across  research  programs.  Finally,  the  lack  of  consensus 
across  fields  ultimately  limits  the  ability  of  researchers  within  each  field  to  share  their  techniques, 
insights,  and  research  results  with  researchers  from  other  fields 

Staggers  and  Norcio  (1993)  looked  at  the  role  of  mental  models  in  human  computer  interactions. 
Though  some  argue  that  the  term  “mental  models”  has  become  meaningless.  Staggers  and  Norcio 
argue  that  they  have  in  fact  become  too  meaningful.  Without  precise  definitions,  the  terms  are 
meaningful  on  intuitive  appeal,  because  they  mean  whatever  readers  want  them  to  mean.  The  only 
basic  requirement  is  that  readers  adopt  the  premise  of  the  mind  operating  in  a  symbolic  fashion. 
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This  raises  the  question  of  whether  researchers  using  different  approaches  are  actually  using  the 
same  underlying  construct  when  they  talk  about  mental  models. 

Beyond  using  the  same  term  (“mental  models”),  the  relationship  between  the  smaller  scale  mental 
models  that  are  the  focus  of  Johnson-Laird’s  reasoning  research  and  the  larger  picture  mental 
models  that  play  a  role  in  Endsley’s  situational  awareness  is  not  clear  to  the  authors.  At  issue  is 
whether  or  not  the  different  mental  models  are  in  fact  the  same  cognitive  construct,  or  at  least 
based  on  similar  cognitive  constructs.  This  has  simply  not  been  explored  in  the  literature. 
Characteristics  from  each  of  the  approaches  should  be  empirically  compared  to  determine  whether 
the  two  (or  more)  theories  can  co-exist,  which  would  support  the  idea  that  the  different  types  of 
mental  models  are  cognitively  similar  (or  not).  The  current  report  will  assume  that  there  is  overlap 
between  the  theories,  focusing  on  common  traits  across  approaches. 

4.2.6.2  Measuring  mental  models 

Endlsey  (2000a)  stated  that  the  utility  of  mental  models  “has  been  limited  by  an  inability  to  reliably 
extract  and  represent  mental  models  in  applied  situations.  The  field  of  cognitive  psychology  may 
be  a  long  time  in  answering  the  basic  scientific  question:  do  people  really  store  knowledge  this 
way?”  (Endsley,  2000a,  p.  5).  One  reason  why  it  is  difficult  to  measure  mental  models  is  because 
an  observer’s  knowledge  of  a  mental  model  is  deeply  embedded  within  the  observer,  making  the 
model  difficult  to  verbalize  (Endsley,  2000a).  Regardless  of  this  difficulty,  researchers  have  used  a 
number  of  methods  to  try  and  measure  mental  models,  including: 

•  Inferring  characteristics  via  empirical  study.  For  example,  the  traditional  psychological 
approach  to  mental  models  research  is  to  use  experimental  methods  to  infer  the 
characteristics  of  models.  Such  methods  provide  only  indirect  insights  into  the  form  and 
structure  of  mental  models;  however,  once  externalized,  changes  in  these  extemalizations 
are  interpreted  as  changes  in  the  underlying  conceptual  systems  (Rouse  &  Morris,  1986). 
The  creation  of  concept  maps  meant  to  represent  externalized  mental  models  is  a  common 
format  (Johnson  et  al,  2006). 

•  Empirical  modelling.  Empirical  modelling  involves  algorithmically  identifying  the  relation 
between  a  person’s  observations  and  subsequent  actions  (Rouse  &  Morris,  1986). 

•  Analytical  modelling.  Analytical  modelling  involves  the  use  of  available  theory  and  data  to 
formulate  assumptions  about  the  form,  structure,  and  parameters  of  mental  models  for 
particular  tasks  (Rouse  &  Morris,  1986). 

•  Verbal/written  reports.  When  using  verbal  or  written  reports,  researchers  simply  ask 
participants  about  their  mental  models  (Rouse  &  Morris,  1986). 

•  Cognitive  task  analysis.  Structured  interviews,  observation,  and  experiment-like  methods 
to  elicit  data  that  are  analyzed  and  represented  as  a  description  of  how  people  think  about 
various  activities  (Klein  &  Hoffman,  2008). 

Regardless  of  the  method  used.  Rouse  and  Morris  (1986)  note  that  “. .  .the  possibility  of  totally 
‘capturing’  the  mental  model  is  rather  remote.  This  is  partly  due  to  the  high  probability  that  a 
mental  model  is  not  a  static  entity  having  only  a  single  form”  (p.  353). 

Rouse  and  Morris  (1986)  further  note  that  it  is  difficult  to  measure  mental  models  because 
researchers’  conceptualization  of  a  mental  model  is  influenced  by  the  domain  being  investigated  as 
well  as  their  background  (e.g.,  psychology  vs.  engineering).  Furthermore,  mental  models  are 
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continually  changing  and  efforts  to  elicit,  measure,  or  describe  them  can  themselves  induce 
changes  in  mental  models  (Klein  &  Hoffman,  2008). 

To  deal  with  such  issues.  Rouse  and  Morris  (1986)  conceptualized  differences  in  mental  models  in 
terms  of  distinctions  among  domains  along  two  dimensions:  the  nature  of  model  manipulation  and 
the  level  of  behavioural  discretion  (see  Figure  9). 
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Figure  9:  Distinctions  among  domains  (Rouse  &  Morris,  1986) 

Along  the  horizontal  axis  is  the  nature  of  model  manipulation.  This  can  range  from  implicit  (i.e., 
the  person  is  not  aware  of  manipulating  her  mental  model)  to  explicit  (i.e.,  the  person  is  aware  that 
she  is  manipulating  her  mental  model).  On  the  vertical  axis,  behavioural  discretion  relates  to 
whether  a  person’s  behaviour  is  a  matter  of  choice  or  is  dictated  by  the  task.  For  instance,  neural 
information  processing  is  unlikely  to  be  discretionary  whereas  tasks  related  to  decision-making  and 
problem  solving  are  likely  to  be  more  discretionary.  Behavioural  discretion,  then,  can  range  from 
none  to  full. 

Distinctions  in  Figure  9  provide  a  basis  for  explaining  methodological  differences  among  domains 
(Rouse  &  Morris,  1986).  First,  inferential  methods  (e.g.,  empirical  modelling,  analytical  modelling) 
tend  to  yield  more  accurate  descriptions  when  there  is  little  discretion  (i.e.  situations  that  fall  into 
the  bottom  portion  of  the  graph)  because  the  conceptualization  can  be  based  on  external 
environmental  or  organizational  constraints.  Second,  verbalization  methods  (i.e.,  interviews. 
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surveys)  provide  more  appropriate  descriptions  when  there  is  explicit  manipulation.  This  is  because 
the  need  for  explicit  manipulation  (i.e.  right  side  of  the  graph)  may  result  in  verbalization  being  a 
natural  part  of  the  task.  Domains  toward  the  upper  left-hand  part  of  Figure  9  are  likely  to  present 
methodological  difficulties  in  the  sense  that  mental  models  will  be  elusive. 

With  respect  to  dynamic  decision-making  tasks,  DDM  is  likely  closer  to  ‘full’  in  terms  of  the  level 
of  behavioural  discretion  that  is  involved  (vertical  axis).  As  stated  in  the  above  explanation,  tasks 
related  to  dynamic  decision  making  are  high  on  this  spectrum.  Flowever,  its  placement  on  the 
nature  of  model  manipulation  (horizontal  axis)  is  more  problematic.  To  some  degree  DDM  spans 
this  dimension,  as  mental  models  appear  to  be  relied  upon  both  consciously  as  well  as 
unconsciously.  Thus,  depending  on  the  decision  maker,  the  task  and  the  environment  DDM  would 
sometimes  be  in  the  upper  right  of  the  table.  In  these  circumstances,  the  nature  of  mental  models  in 
dynamic  decision-making  should  be  possible  to  measure  through  verbalization  methods. 
Circumstances  when  mental  model  manipulation  is  more  unconscious  would  put  mental  models  in 
the  upper  left,  thus  limiting  effective  measurement  techniques.  These  results  are  consistent  with 
the  literature  finding  mental  models  hard  to  capture.  Further  study  related  to  the  consciousness  of 
mental  model  manipulation  is  needed. 

Canas,  Antoli,  and  Quesada  (2001)  conducted  research  to  better  understand  the  methods  used  to 
measure  mental  models.  They  argued  that  the  problem  with  using  techniques  such  as  knowledge 
elicitation  to  capture  the  knowledge  contained  in  mental  models  is  that  the  model  is  only  a 
simulation  of  the  process  is  being  measured.  People  give  their  responses  based  on  a  simulation  of 
the  real  task,  and  only  the  knowledge  used  in  that  particular  instance  is  elicited.  Therefore,  such 
elicitation  techniques  do  not  gather  a  complete  account  of  all  information  within  Long  Term 
Memory  (LTM),  but  access  only  the  knowledge  in  Working  Memory  (WM).  They  argue  that: 

“if  a  mental  model  is  a  knowledge  structure  that  is  simulated  in  WM,  we  must  speak  of  the 
Mental  Model  as  a  process  and  as  the  result  of  that  process.  When  we  measure  the  Mental 
Model  we  are  measuring  the  result  of  the  simulation  process.  This  result  we  take  as  a 
reflection  of  the  knowledge  structure  that  is  stored  in  LTM...  However,  the  simulation  is 
accomplished  by  selecting  the  part  of  the  permanent  knowledge  that  is  relevant  for  the  task. 
That  is  to  say,  not  all  the  knowledge  is  selected.  The  part  that  is  selected  will  depend  on  the 
task,  the  context,  the  intentions,  etc.  It  is  also  possible  that  the  mental  model,  as  measured, 
might  be  affected  not  only  by  selection  but  also  by  transformations  performed  on  the 
knowledge  in  order  to  comply  with  the  elicitation  task”  (p.  27). 

To  better  understand  this  issue,  Canas  et  al.  (2001)  conducted  a  series  of  experiments  to 
demonstrate  that  task  characteristics  affect  simulations  in  WM  and,  consequently,  what  is  inferred 
about  knowledge  of  the  system.  Their  hypothesis  was  that  a  “. .  .mental  model  is  a  dynamic 
representation  created  in  working  memory  by  combining  information  stored  in  long-term  memory 
and  characteristics  extracted  from  the  environment”  (p.  28).  Results  from  their  research  suggest 
that  variables  introduced  to  impact  WM  had  an  impact  on  participants’  execution  of  the  elicitation 
task.  All  participants  learned  to  interact  with  the  system  and  could  answer  questions  about  the 
system,  but  their  ratings  about  relationships  within  the  system  were  impacted  by  what  happened 
during  the  elicitation  tasks.  Subsequently,  Canas  et  al.  (2001)  concluded  that  elicitation  tasks  do,  in 
fact,  measure  the  content  of  WM,  not  the  knowledge  stored  in  LTM. 

Thus  it  appears  that  trying  to  measure  mental  models  is  a  complicated  task.  Our  review  suggests 
that  although  there  are  a  number  of  approaches  to  capturing  mental  models,  their  use  is  somewhat 
dependent  on  the  field  of  study  of  the  researchers,  and  the  assumptions  that  they  make  about  what 
mental  models  actually  are.  Although  there  are  concerns  that  such  methodologies  do  not  capture 
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mental  models  accurately,  research  by  Canas  et  al.  (2001)  suggests  that  some  aspects  of  mental 
models  can  be  captured  by  researchers.  Furthermore,  Klein  and  Floffman  (2008)  argue  that 
studying  mental  models  can  be  manageable  if  we  focus  our  inquiry  on  the  knowledge  that  people 
have  about  specific  relationships  (e.g.,  cognitive  task  analysis). 

4.3  Using  Mental  Models 

It  is  also  critical  to  consider  research  and  theory  related  to  how  mental  models  are  actually  used. 
This  sections  explores  the  available  literature  relevant  to  how  mental  models  are  formed  and 
updated,  provides  specific  examples  of  mental  model  applications,  and  considers  the  benefits  and 
challenges  of  using  mental  models. 

4.3.1  Forming  and  Updating  Mental  Models 

Although  there  is  a  vast  literature  on  mental  models,  very  little  literature  exists  that  provides  a 
description  of  the  cognitive  processes  involved  when  mental  models  are  formed.  Research  that 
does  exist  suggests  that  mental  models  are  built  from  the  knowledge  required  to  pursue  a  goal  and 
the  data  perceived  from  the  environment  (Besnard,  Greathead,  &  Baxter,  2004).  That  is,  mental 
models  are  formed  after  an  observer  scans  the  environment  of  interest,  selects  cues  from  this 
environment,  and  subjectively  interprets  the  cues  (Richardson  et  al.,  1994).  A  similar  description 
argues  that  the  information  perceived  and  how  it  is  interpreted  is  guided  by  pre-existing  concepts 
(Bryant,  2004).  Such  pre-existing  concepts,  or  mental  representations,  guide  perception  and 
information  gathering  activities.  These  concepts  essentially  tell  us  how  to  observe  the  world, 
discern  what  is  and  is  not  relevant,  and  relate  observed  phenomena  to  our  goals  (Bryant,  2004). 

Denzau  and  Roy  (2005)  argue  that  mental  models  are  contextual,  and  are  learned  in  relationship  to 
what  we  know,  believe,  and  fear.  We  then  assimilate  knowledge  into  our  mental  models  based  on 
these  beliefs.  For  example,  Denzau  and  Roy  (2005)  stress  the  importance  of  one’s  cultural 
background  in  shaping  one’s  mental  models: 

Under  conditions  of  uncertainty,  individuals’  interpretations  of  their  environment  will 
reflect  the  learning  that  they  have  undergone.  Individuals  with  common  cultural 
backgrounds  and  experiences  will  share  reasonable  convergent  mental  models,  ideologies 
and  institutions  and  individuals  with  different  learning  experiences  (both  cultural  and 
environmental)  will  have  different  theories  (models,  ideologies)  to  interpret  that 
environment.  Moreover,  the  information  feedback  from  their  choices  is  not  sufficient  to 
lead  to  convergence  of  competing  interpretations  of  reality.  In  such  cases  multiple 
equilibria  will  result  (p.  2). 

It  is  also  important  to  note  that,  when  selecting  cues  from  the  environment,  human  decision  makers 
need  to  be  economical  in  their  use  of  data  (Bryant,  2004).  It  is  not  the  amount  of  data  that  is  the 
paramount  factor,  but  rather  the  informativeness  of  the  data.  The  informativeness  allows  the 
decision  maker  to  evaluate  the  validity  of  his/her  mental  models  (Bryant,  2004).  This  is  consistent 
with  Gonzalez  (2005)  who  found  that  the  absence  of  irrelevant  information  helps  participants  in 
DDM  tasks  (see  section  3.2. 1  for  more  on  this). 

Denzau  and  Roy  (2005)  state  that  the  best  conditions  to  form  mental  models  are  those  in  which  an 
observer  is  observing  a  relatively  simple  situation,  there  is  good  quality  information,  there  is 
frequent  feedback,  and  the  observer  is  sufficiently  motivated  to  incur  the  costs  of  learning.  It  is 
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under  these  circumstances  that  an  observer  is  most  likely  to  learn  the  correct  mental  model  through 
which  to  see  the  world. 

When  discussing  the  formation  of  mental  models,  a  distinction  must  be  made  between  forming  a 
new  model  and  forming  the  parameters  of  an  existing  model  (Denzau  &  Roy,  2005).  Research  has 
shown  that  observers  find  it  easier  to  incorporate  new  information  into  existing  models  than  to 
form  new  models  (Staggers  &  Norcio,  1993).  This  could  be  because  forming  new  models  depends 
on  a  number  of  factors;  such  as  (a)  the  motivation  of  the  user  (b)  the  complexity  of  the  system  and 
(c)  the  complexity  of  the  task.  Whereas  observers  might  be  motivated  to  form  a  new  model  because 
of  the  perceived  need  for  information  in  the  system,  the  complexity  of  a  task  or  system  might 
inhibit  the  formation  of  new  mental  models. 

According  to  Bower  and  Morrow  (1990),  the  power  of  narrative  can  promote  the  formation  of 
mental  models.  In  a  paper  describing  the  role  of  mental  models  in  narrative  comprehension.  Bower 
and  Morrow  (1990)  argue  that  the  basis  of  language  comprehension  is  a  reader’s  ability  to 
construct  mental  models  of  the  situation  being  described.  Understanding  a  narrative  involves: 

1)  translating  text  to  underlying  conceptual  propositions,  and 

2)  using  world  knowledge  to  identify  referents  of  the  text’s  concepts,  linking  expressions 
referring  to  the  same  entity,  drawing  inferences,  and  knitting  together  causal  relations 
between  action  sequences. 

As  noted  in  section  4.2. 3.2,  comprehension  of  a  text  requires  the  construction  of  a  mental 
representation  of  what  the  text  is  about:  a  situation  model.  The  situation  model  is  used  to  interpret 
and  evaluate  later  statements  in  the  text.  Much  of  what  a  dynamic  decision  maker  learns  about  the 
world  is  through  text  (e.g.  emails,  articles,  powerpoint  presentations,  reports,  etc).  While  DDM  is 
not  explicitly  formed  through  narrative  text,  there  are  lessons  here  that  are  applicable. 

A  key  intuition  often  associated  with  mental  models  is  that  they  are  ‘runnable’,  i.e.  there  is  a  sense 
of  deriving  answers  via  mental  simulation  rather  than  logical  reasoning.  Many  studies  have  looked 
at  the  idea  that  people  learn  how  to  make  mental  simulations  of  phenomena  either  in  a  series  of 
dynamic  images  in  the  minds’  eye  or  in  more  abstract  mental  models  (Johnson-Laird,  2005).  Thus, 
running  the  model  is  the  process  of  manipulating  the  mental  model  in  one’s  own  head  to  test 
hypotheses  about  the  reality  the  model  represents.  By  using  rules  and  structural  relationships 
among  objects,  the  model  is  run  in  a  qualitative  sense  as  well  as  in  a  more  cause-and-effect  mode 
(Staggers  &  Norcio,  1993).  While  this  is  cited  as  a  common  function  of  mental  models,  some 
argue  that  people’s  abilities  to  “run”  their  models  are  severely  limited  (Norman,  1983;  as  cited  in 
Staggers  &  Norcio,  1993).  Possible  causes  for  errors  include  limited  cognitive  resources, 
erroneous  mental  models  and  mental  models  that  are  not  complete  enough  to  run.  It  is  plausible 
that  a  concise  and  coherent  mental  model  can  be  used  to  make  accurate  predictions  about  the  real 
world,  assuming  there  are  only  a  limited  number  of  variables  to  track. 

4.3.2  Application  of  Mental  Models 

A  number  of  examples  found  in  literature  seem  to  provide  important  information  relevant  to  how 
mental  models  are  used  by  people  in  settings  such  as  military  decision  making,  as  well  as.general 
problem  solving  and  decision-making. 

The  Observe-Orient-Decide-Act  (OODA)  Loop  has  been  a  popular  descriptive  model  of  reasoning 
and  decision-making  within  military  circles  for  roughly  50  years  (Bryant,  2004).  Within  this 
model,  friendly  and  enemy  forces  are  seen  as  being  in  competing  cycles  of  decision  processes.  This 
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has  allowed  military  theorists  to  identify  ways  to  speed  up  enemy  deeision-making  while 
interfering  with  and  slowing  down  enemy  deeision-making.  The  key  assumption  is  that  eompleting 
one’s  own  decision  cycle  faster  than  one’s  opponent  will  yield  ever-increasing  advantages  in 
command  and  control  (C2)  effectiveness,  which  will,  in  turn,  yield  greater  battle  success  (Alex, 
2000;  as  cited  in  Bryant,  2004). 

Bryant  (2004)  has  criticized  the  OODA  loop  as  an  outdated  model  of  cognition.  Despite  its 
intuitive  appeal,  he  argues  that  the  processes  that  people  use  to  seek  and  use  information  and  to 
generate  and  select  courses  of  action  are  not  explicit  in  this  model.  Moreover,  it  portrays  command 
decision-making  as  reactive  rather  than  proactive.  It  also  overlooks  the  fact  that  decision-making 
has  both  “bottom-up”  and  “top  down”  components,  and  that  critical  thinking  and  constructive 
processes  such  as  mental  models  help  to  guide  human  decision-making.  He  proposes  an  alternative 
descriptive  model,  the  Critique,  Explore,  Compare,  Adapt  (CECA)  Eoop.  This  model  is  predicated 
on  two  mental  representations.  The  first  is  the  conceptual  model,  which  is  derived  through  the 
operational  planning  process.  The  second  is  the  situation  model,  which  captures  the  state  of  the 
battle  space  at  a  given  moment  in  time.  The  conceptual  model  is  “how  you  want  it  to  be”.  It  must 
be  goal  directed  and  describe  the  states  of  the  battle  space  one  wants  to  achieve  across  a  specified 
period  of  time.  This  is  much  more  important  than  describing  what  actions  one  believes  should  be 
performed  to  meet  operational  goals.  The  situation  model  is  a  representation  of  “how  it  currently 
is”  (Bryant,  2004).  The  CECA  model  is  shown  in  Figure  10. 


Figure  10:  CECA  Loop  (Bryant,  2004) 

The  key  to  effective  information  management  depends  on  minimizing  attention  devoted  to 
information  that  does  not  have  the  potential  to  invalidate  the  conceptual  model.  Disconfirmatory 
evidence,  which  can  indicate  ways  in  which  the  conceptual  model  is  not  an  accurate  representation 
of  the  situation,  is  more  valuable  than  confirmatory  evidence  (Bryant,  2004).  Acknowledging  this 
disconfirmation  evidence  can  be  a  severe  challenge  for  decision  makers  however  (see  Confirmation 
Bias  the  next  section). 
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The  Critique  phase  involves  questioning  the  conceptual  model  (“how  you  want  it  to  be”)  to  identify 
critical  aspects  that  if  invalidated  would  render  the  plan  for  the  operation  untenable  in  some 
respect.  The  Explore  phase  comprises  the  active  and  passive  collection  of  data  from  the  battle 
space.  The  Compare  phase  involved  comparing  the  “how  it  currently  is”  with  “how  you  want  it  to 
be”  and  identifying  any  elements  that  are  inconsistent  with  the  current  situation.  The  Adapt  phase 
involves  the  decision  maker  determining  what  to  do  in  response  to  the  inconsistencies.  All  actions 
are  driven  by  the  conceptual  model,  which  lays  out  the  rationale  for  each  action  the  own  force 
might  take  (Bryant,  2004),  consistent  with  Richardson  et  al.’s  (1994)  means/end  models  (discussed 
in  section  4.2.4).  The  CECA  model  is  a  good  example  of  how  the  concept  of  mental  models  has 
been  used  to  elucidate  complex  decision-making  processes. 

Other  research  has  also  explored  the  use  of  mental  models  in  military  contexts,  in  relation  to  the 
decision-making  skills  of  novice  and  experienced  sea-combat  trainees.  After  observing  sea-combat 
trainees  in  training  simulator  scenarios,  Eipshitz  and  Ben  Shaul  (1994)  noticed  important 
differences  in  how  expert  and  novice  trainees  performed.  They  observed  that  experts  collected 
more  information  about  the  situation  before  making  a  decision,  engaged  in  more  efficient 
information  searches,  “read”  the  situation  more  accurately,  made  fewer  bad  decisions,  and 
communicated  more  frequently  and  elaborately  with  friendly  units.  To  make  sense  of  these 
differences  between  the  expert  and  novice  trainees,  Eipshitz  and  Ben  Shaul  were  interested  in 
understanding  differences  between  the  experts’  and  the  novices’  mental  models.  They 
conceptualized  mental  models  is  “specific  situation  representations”  (p.  293)  that  drive  decision¬ 
making.  However,  they  also  argued  that  “mental  models  [. . .]  are  labile  entities  that  are  constructed 
and  discarded  as  decision-makers  move,  in  time  and  space,  from  one  situation  to  another  (p.  298). 
Therefore,  Eipshitz  and  Ben  Shaul  inquired  into  differences  in  experts’  and  novices’  schemata, 
which  are  “abstract  cognitive  structures  that  guide  the  construction  of  mental  models”  (p.  293),  to 
understand  why  the  two  groups  constructed  different  mental  models  in  identical  situations.  As 
schemata  guide  the  construction  of  mental  models  through  information  search  and  interpretation,  it 
is  important  to  have  adequate  schemata  in  order  to  have  an  accurate  mental  model.  Eipshitz  and 
Ben  Shaul  argue  that  experts  are  more  likely  to  have  superior  schemata. 

To  understand  why  the  two  groups  constructed  different  mental  models,  Eipshitz  and  Ben  Shaul 
(1997)  turned  to  Klein’s  (1993;  as  cited  in  Eipshitz  and  Ben  Shaul,  1997)  Recognition-Primed 
Decision-making  (RPD)  model.  Although  applicable  to  the  training  scenario,  Eipshitz  and  Ben 
Shaul  ( 1 997)  did  not  feel  that  the  RPD  model  could  account  for  their  observation  that  experts 
conduct  more  extensive  and  efficient  information  searches  before  making  decision  than  novices. 
Therefore,  the  researchers  added  schemata  to  the  RPD  model. 
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Figure  11.  RPD  model  as  schemata-driven  mental  modelling  (Lipshitz  &  Ben  Shaul, 

1997) 


Lipshitz  and  Ben  Shaul  (1997)  argue  that  adding  schemata  to  the  RPD  model  explains  the 
differences  between  expert  and  novice  trainees  that  the  RPD  model  could  not  explain  otherwise.  In 
turn,  adding  schemata  to  the  RPD  model,  they  argue,  makes  the  RPD  model  more  powerful. 

Mental  models  have  also  been  used  to  try  to  understand  problem  solving  in  general.  For  example, 
Kolkman,  Kok,  and  van  der  Veen  (2005)  included  mental  models  as  a  key  factor  in  the  problem 
solving  cycle  (see  Figure  12). 


Figure  12.  Steps  of  the  problem  solving  cycle  (Kolkman,  Kok,  &  van  der  Veen, 

2005). 

Kolkman  et  al.  (2005)  state  that  the  meaning  of  information  is  socially  constructed  and  guided  by 
different  frames  of  perception.  Such  frames  are  one’s  underlying  structures  of  belief,  perception, 
and  appreciation.  The  steps  of  the  problem  solving  cycle  are  influenced  by  the  frame  of  perception 
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and,  indirectly,  by  the  mental  model.  The  mental  model  acts  as  a  ‘fdter’  that  selects  information 
from  the  ‘real  world’  to  be  used  in  the  frame.  Mental  models  determine  what  data  the  observer 
perceives  from  the  outside  world  and  what  knowledge  the  observer  derives  from  it. 

Mental  models  have  also  been  used  to  shed  light  on  dynamic  decision-making  within  the  health 
care  industry.  Shultz,  Dutta,  and  Johnson  (2000)  were  interested  in  understanding  the  thinking  of 
health  care  executives  as  they  engage  in  DDM  tasks.  In  particular,  they  were  interested  in 
understanding  relationships  between  mental  models,  use  of  feedback/feedforward  decision  control 
strategies,  and  changing  conditions  in  the  decision  environment  for  health  care  executives. 

Shultz  et  al.  (2000)  had  senior  healthcare  managers  assume  the  role  of  a  Chief  Executive  Officer  of  an 
integrated  health  care  organization  and  make  resource  allocation  decisions  over  a  simulated  20  year 
period  of  time.  The  researchers  found  that  financial  models  and  feedforward  control  (as  opposed  to 
feedback  control)  were  characteristics  of  successfril  participants.  They  further  found  that  health  care 
executives  tended  to  have  mental  models  related  to  the  financial  viability  of  organizations  rather  than 
mental  models  related  to  quality  of  care.  Finally,  the  more  turbulent  the  changing  environmental 
conditions  were,  the  less  successfril  participants  were  in  completing  the  simulation. 

Wilson  and  Rutherford  (1989)  note  that  mental  models  are  of  central  importance  within  human 
factors  research.  They  note  that  the  concept  of  mental  models  is  often  used  or  invoked  within 
human  factors  research  in  order  to: 

•  Guide  general  design  (e.g.,  “know  the  user’s  mental  model’’); 

•  To  summon  the  idea  of  a  “picture  in  the  mind’’; 

•  As  experimenter-created  or  hypothesized  models  to  measure  task  performance; 

•  Estimate  the  complexity  of  system  use  to  enhance  system  design; 

•  To  explain  and  describe  behaviour; 

•  Used  as  a  design  tool. 

Of  course,  the  concept  of  mental  models  has  continued  to  receive  interest  and  attention  because  of 
the  perceived  benefits  that  they  offer.  Although  mental  models  are  argued  to  have  a  number  of 
positive  benefits,  they  are  argued  to  be  most  useful  in  simple  rather  than  complex  environments, 
and  in  more  familiar  environments.  Relatively  simple  environments/systems  would  require 
observers  to  perceive  only  a  limited  amount  of  information  to  form  their  mental  models  (Denzau 
and  Roy,  2005).  Simpler  systems  would  also  reduce  the  number  of  mental  models  that  an 
individual  is  required  to  maintain  (Johnson-Eaird,  2001).  Both  these  factors  ease  the  mental 
workload  in  working  memory  required  in  using  mental  models,  thus  improving  their  efficacy.  This 
characteristic  is  not  consistent  with  dynamic  decision-making  environments  where  the  systems  are 
complex,  full  of  interdependencies  and  uncertain. 

As  noted,  mental  models  are  also  generally  acknowledged  to  be  most  helpful  in  relation  to  familiar 
environments.  Forming  a  mental  model  is  not  automatic.  It  takes  effort  on  the  part  of  an  individual 
in  order  to  select  from  the  environment  relevant  data,  integrate  it  with  other  information,  and  create 
some  form  of  coherence.  Once  a  mental  model  is  formed,  it  is  relatively  simple  to  add  information 
to  that  model  (whether  that  new  information  improves  the  model  or  not)  compared  to  the  effort 
required  to  create  a  new  mental  model. 

Our  review  showed  strong  support  for  the  potential  importance  of  mental  models.  In  short,  they 
help  to  simplify  one’s  environment,  and  can  assist  problem  solving  and  decision-making  because 
they  allow  “short-cuts”.  Having  a  well- formed  mental  model  can  allow  us  to  walk  into  a  familiar 
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situation,  and  to  know  how  to  react,  because  our  mental  model  can  guide  our  behaviour.  Mental 
models  are  parsimonious:  often  people  do  extra  physical  operations  rather  than  the  mental  planning 
that  would  allow  them  to  avoid  those  actions;  they  are  willing  to  trade-off  extra  physical  action  for 
reduced  mental  complexity.  This  is  especially  true  when  the  extra  actions  allow  one  simplified  rule 
to  apply  to  a  variety  of  devices,  thus  minimizing  the  chances  for  confusions  (Norman,  1983;  as 
cited  in  Staggers  &  Norcio,  1993). 

Just  as  mental  models  are  argued  to  have  adaptive  value,  however,  they  are  also  noted  to  have  a 
range  of  shortcomings  (e.g.,  Doyle  and  Ford,  1998).  For  example,  mental  models  are  not 
necessarily  either  accurate  or  complete.  Some  of  the  prominent  shortcomings  noted  in  the  literature 
are  as  follows: 

Limited  cognitive  capacity.  Forrester  (1994;  as  cited  in  Doyle  &  Ford,  1998)  argued  that  people  can 
only  properly  relate  a  limited  number  of  variables.  In  fact,  he  states  that  even  a  skilled  investigator 
is  unable  to  anticipate  the  dynamic  behaviour  of  a  simple  information-feedback  system  including 
more  than  five  or  six  variables.  This  suggests  that  people  are  unable  to  create  accurate  mental 
models  of  highly  complex  systems. 

Confirmation  bias.  Mental  models  have  also  been  argued  to  be  subject  to  confirmation  biases. 
Besnard,  Greathead,  and  Baxter  (2004)  state  that  the  “weakness  of  mental  models  lies  in  their  poor 
requirements  in  terms  of  validity:  if  the  environmental  stream  of  data  is  consistent  with  the 
operator’s  expectations,  that  is  enough  for  the  operator  to  continue  regarding  the  mental  model  as 
valid”  (p.  119).  Overall,  then,  we  are  often  motivated  to  see  events  as  being  in  accordance  with  our 
beliefs  and  expectations  rather  than  being  inconsistent  with  them.  This  can  lead  to  errors  and  biased 
reasoning  processes,  because  we  will  tend  to  treat  our  experiences  as  evidence  that  our  beliefs  are 
correct  and,  erroneously  believe  that  we  have  understood  the  problem  at  hand.  Mental  models  can 
be  conceptualized  as  a  consistent  set  of  expectations  about  a  specific  event  or  entity.  As  such, 
incoming  information  that  is  inconsistent  with  our  mental  models  may  well  receive  less  weight 
and/or  attention  than  it  should.  Therefore,  such  “co-occurring  events  can  seriously  disrupt  situation 
awareness  when  humans  are  using  mental  models  that  are  highly  discrepant  to  reality  but 
nonetheless  trusted”  (Besnard,  Greathead,  &  Baxter,  2004,  p.  118). 

Incomplete  information.  Mental  models  have  been  described  as  being  characterized  by  their 
incompleteness  (Besnard,  Greathead,  &  Baxter,  2004).  This  is  because  mental  models  have  a 
limited  scope  and  their  content  is  only  a  partial  representation  of  the  environment,  because  of  the 
limitations  of  human  cognition  (e.g.,  people  forget  the  details  of  the  system  (Norman,  1983;  as 
cited  in  Staggers  &  Norcio,  1993).  Similarly,  the  number  of  elements  that  can  be  held  in  working 
memory  is  also  limited.  The  resulting  world  view  is  one  in  which  the  essential  features  of  a 
problem  are  overemphasized  and  peripheral  data  can  be  overlooked.  Consistent  with  confirmation 
bias  described  in  the  previous  paragraph,  mental  models  can  also  be  inaccurate  because  observers 
sometimes  ignore  new  information  and  continue  to  interpret  the  situation  in  light  of  their  formed 
but  incorrect  mental  models.  As  noted  by  Moray  (1987;  cited  in  Staggers  &  Norcio,  1993), 
observers  can  have  a  tendency  to  become  rigidly  stuck  in  one  mode  and  not  update  their  mental 
model  in  the  face  of  new  information. 

System  errors.  There  are  systemic  errors  in  reasoning  with  mental  models.  People  systematically 
overlook  possible  models  of  individual  premises  (e.g.  they  often  incorrectly  treat  all  x  are  y  as 
referring  to  two  co-extensive  sets).  Situations  that  call  for  a  single  mental  model  are  reliably  easier 
that  those  that  call  for  more  than  one  model  (Johnson-Laird,  2005). 
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Erroneous.  Mental  models  have  been  criticized  as  often  being  inaccurate  or  altogether  erroneous 
(Klein  &  Hoffman,  2008).  Research  on  laypersons’  and  students’  mental  models  have  found  them 
to  be  “fuzzy,  implicit,  mostly  wrong,  vastly  simplified,  dynamically  deficient,  broad,  amorphous’’ 
(Klein  &  Hoffman,  2008,  p.61).  Such  erroneous  findings  have  been  found  in  research  conducted  to 
understand  mental  models  on  topics  such  as  the  causes  of  global  warming,  the  workings  of 
computers,  and  understanding  risk.  This  is  consistent  with  Johnson-Laird’s  (2005)  findings  that 
people  make  systemic  errors  in  their  models  (e.g.  they  often  incorrectly  treat  all  x  are  y  as  referring 
to  two  co-extensive  sets). 

Overall,  then,  despite  their  potential  contribution  to  human  performance,  mental  models  are  also 
subject  to  a  range  of  shortcomings  that  impact  the  accuracy  of  mental  models  are  that  people  have  a 
limited  cognitive  capacity,  that  humans  often  make  erroneous  causal  links,  that  models  are  often 
based  on  incomplete  information,  and  that  systemic  errors  in  reasoning  impact  mental  models. 
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5.  Mental  Models  and  Dynamic  Decision¬ 
making 


Having  considered  the  concept  of  mental  models  from  a  range  of  perspectives,  then,  the  goal  of  this 
section  is  to  explore  the  usefulness  of  the  mental  model  concept  within  the  domain  of  dynamic 
decision-making.  This  section  considers  potential  linkages  between  specific  mental  model 
approaches  and  DDM.  The  section  that  follows  considers  the  requirements  for  optimally  applying 
mental  models  while  performing  dynamic  decisions. 

5.1  Approaches  to  Mental  Models  and  DDM 

Previous  chapters  explored  the  nature  of  DDM  and  the  various  approaches  to  mental  models 
prominent  in  the  literature.  The  goal  of  this  section  is  to  understand  the  extent  to  which  DDM  could 
be  supported  by  each  of  the  unique  approaches  to  understanding  mental  models.  As  alluded  to 
already,  there  are  a  number  of  discrepancies  between  DDM  environments  and  the  context  in  which 
mental  models  are  best  applied.  In  general,  the  goal  of  the  analysis  was  to  be  as  inclusive  as 
possible.  That  meant  that  instead  of  ruling  out  perspectives  at  first  signs  of  a  contradiction,  we 
instead  chose  to  review  the  entire  perspective  and  try  to  tease  out  aspects  that  could  be  applicable 
in  a  DDM  environment. 

5.1.1  Propositional  Models 

While  it  is  agreed  that  propositional  models  are  based  in  psychology,  it  is  difficult  to  see  any 
substantive  symmetries  between  the  Johnson-Laird  approach  to  mental  models  and  the 
requirements  of  DDM.  Instead,  two  specific  aspects  of  propositional  models  will  be  discussed 
below.  First,  differences  in  the  contexts  of  Johnson-Laird’s  studies  versus  DDM  will  be  discussed. 
Second,  the  extent  to  which  deductive  reasoning  is  applied  by  decision  makers  in  dynamic 
environments  will  be  explored. 

5. 1. 1. 1  The  Context  of  Propositional  Models  and  DDM 

In  Johnson-Laird’s  studies,  reasoners  are  presented  with  concrete,  static  relationship  scenarios. 
While  logically  complex,  they  do  not  change  over  time,  involve  no  interdependence  between 
decision  makers,  and  are  relatively  well  structured.  These  situations  are  in  high  contradiction  to  the 
world  in  which  dynamic  decision-making  takes  place.  While  this  does  not  mean  that  it  is 
impossible  for  propositional  models  to  play  a  part  in  DDM,  it  does  limit  their  applicability. 

Casting  human  decision-making  as  requiring  a  high  number  of  complex  mental  models  seems  at 
odds  with  the  notion  of  proposition  models.  In  fact,  one  of  the  conclusions  from  the  reasoning 
studies  was  that  reasoners  perform  better  when  there  is  only  one  mental  model  required  where  each 
mental  model  represents  one  set  of  possibilities.  If  applied  to  the  DDM  domain,  this  would  mean 
that  the  decision  maker  creates  a  single  model  for  each  and  every  possible  outcome.  Given  the 
sheer  quantity  of  possible  permutations,  on  even  the  simplest  of  dynamic  decision-making  tasks,  it 
is  highly  improbable  that  decision  makers  are  capable  of  doing  this.  Further,  Johnson-Laird  found 
that  the  differences  between  scenarios  requiring  two  and  three  mental  models  were  often  so  small 
that  it  was  unlikely  that  reasoners  constructed  all  three  models. 
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If  decision  makers  are  challenged  when  they  are  required  to  create  more  than  three  mental  models, 
perhaps  this  explains  why  decision  makers  in  DDM  scenarios  sometimes  do  not  fare  well. 

5.1. 1.2  Deductive  versus  Inductive  Reasoning 

Proponents  of  propositional  model  theory  depict  decision  makers  as  cognitively  building  on  given 
parts  of  a  construct  to  create  a  mental  model.  This  mental  model  (or  set  of  assumptions)  is  then 
used  to  deduce  conclusions  about  the  state  of  the  world. 

While  the  role  of  deductive  versus  inductive  reasoning  in  DDM  does  not  appear  to  be  prominent  in 
the  available  literature,  some  inferences  about  their  potential  usage  can  be  made.  There  are  some 
basic  differences  between  deductive  and  inductive  reasoning.  Deductive  reasoning  involves 
deriving  consequences  from  what  is  assumed.  If  the  assumption  is  correct,  then  the  consequence 
will  be  valid.  Inductive  reasoning  is  quite  different.  Inductive  reasoning  involves  inferring  a 
consequence  based  on  multiple  instantiations  of  similar  scenarios  where  those  scenarios  have  led  to 
that  consequence.  Induction  is  not  guaranteed  to  be  correct,  no  matter  if  all  the  previous  scenarios 
are  in  fact  truthful  (e.g.  one  may  conclude  that  all  swans  are  white  as  they  have  only  ever  seen 
white  swans,  but  this  does  not  mean  that  black  swans  do  not  exist).  In  general,  errors  in  reasoning 
occur  two  ways  (1)  in  deduction  where  the  original  assumptions  are  incorrect  or  (2)  in  induction 
where  the  consequence  is  in  fact  not  generalizable. 

Johnson-Laird’s  theories  generally  predict  that  errors  are  caused  by  the  first  error  outlined  above 
(i.e.  errors  in  the  mental  model,  which  represent  the  assumptions  and  lead  to  incorrect 
consequences).  While  this  is  also  logically  applicable  to  DDM,  the  second  error  (i.e.  induction 
errors)  are  also  possible.  This  would  occur  when  a  DDM  decision  maker  incorrectly  believes  that 
one  scenario  is  like  another  scenario  leading  to  incorrectly  assumed  consequences.  Logically,  these 
types  of  errors  would  occur  even  if  the  mental  model  (of  previous  scenarios)  is  correct.  The 
relationship  between  inductive  reasoning  and  mental  models  is  further  explored  by  Holland  et  al 
(1986)  in  their  book  Induction.  They  state  that  “despite  their  inherently  transitory  nature  -  indeed 
because  of  it  -  mental  models  are  the  major  source  of  inductive  change  in  long-term  knowledge 
structures. . .  [as]  model  construction  provides  the  opportunity  for  new  ideas  to  arise  by 
recombination  and  as  a  consequence  of  disconfirmation  of  model-based  predictions.” 

Research  looking  more  into  the  role  of  inductive  versus  deductive  reasoning  and  the  role  in  DDM 
could  prove  interesting  while  also  providing  support  for  or  against  relations  between  propositional 
models  and  mental  models. 

5.1.2  Physical  Systems 

Mental  models  of  physical  systems  are  the  models  people  use  to  conduct  everyday  reasoning  about 
the  physical  world  (Forbus  &  Gentner,  1997;  Centner,  2002).  Such  models  are  said  to  be  important 
for  reasoning  about  complete  physical  systems,  making  and  articulating  predictions  about  the 
world,  and  discovering  causal  explanations  for  what  happens  around  us  (Markham  &  Gentner, 
2001). 

Little  of  the  literature  discussing  mental  models  of  physical  systems  has  been  discussed  in  such  a 
way  that  it  can  be  related  to  DDM.  In  fact,  there  is  a  dearth  of  information  to  explain  how  such 
mental  models  are  formed,  how  they  are  accessed,  or  even  how  they  are  used.  Therefore,  this 
discussion  will  focus  on  particular  information  items  available  about  mental  models  of  physical 
systems  and  how  they  might  be  related  to  DDM.  We  will  also  discuss  key  aspects  of  DDM  and 
how  such  information  is  dealt  with  by  physical  system  models. 
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5. 1.2. 1  Models  of  physical  systems 

Inconsistent  models.  Gentner  (2002)  states  that  people  can  hold  two  or  more  inconsistent  mental 
models  within  the  same  domain.  The  example  given  was  that  of  a  novice  learner  who  may  give  one 
explanation  for  why  a  towel  dries  in  the  sun  and  another  explanation  for  what  causes  a  puddle  of 
water  to  evaporate  without  seeing  any  connection  between  the  two  situations  (Collins  &  Gentner, 
1987;  as  cited  in  Gentner,  2002).  This  begs  the  question:  how  do  dynamic  decision  makers  choose 
between  inconsistent  mental  models  when  making  dynamic  decisions?  It  would  be  interesting  to 
know  what  criteria  are  used  to  decide  which  mental  model  should  be  used  in  DDM  tasks.  We  also 
wonder  how  having  inconsistent  models  impact  on  the  effectiveness  of  DDM. 

Experience.  Markham  and  Gentner  (2001)  suggest  that  people  shift  to  using  learning  rules  rather 
than  mental  models  once  they  have  gained  experience  with  a  domain.  After  repeated  exposure  to 
the  domain,  decision  makers  start  to  associate  certain  patterns  with  certain  outcomes.  Markham 
and  Gentner  (2001)  argue  that  this  leads  experienced  decision  makers  to  access  their  stored 
knowledge  of  the  outcomes  rather  than  to  carry  out  mental  simulations.  This  assertion  is  based  on 
the  assumption  that  using  mental  models  implies  mental  simulation  (or  ‘running’  of  the  mental 
models  to  predict  outcomes  with  different  variables).  They  assert  instead  that  expert  dynamic 
decision  makers  no  longer  use  mental  models  at  all,  but  rather  rely  on  their  heuristics  and  schemas 
to  carry  out  DDM.  This  is  important  research  for  understanding  the  use  of  mental  models  in  DDM, 
especially  in  comparing  the  effectiveness  of  expert  and  novice  decision  makers. 

5.1. 2.2  DDM  variables 

Control  of  the  system.  The  main  aim  of  the  dynamic  decision  maker  is  to  gain  control  of  the 
environment  (Clancy  et  al.,  2003).  However,  there  is  no  discussion  of  the  observer  using  physical 
system  models  to  control  their  environment.  The  literature  simply  states  that  mental  models  are 
used  to  reason  about,  make  predictions  of,  and  discover  causal  explanations  for  the  every  day 
environment  (Gentner,  2002).  This  suggests  that  control  is  either  not  an  issue  when  forming  mental 
models  for  physical  systems  or  that  control  is  an  issue  that  has  not  yet  been  explored  by 
researchers. 

Environment.  Markman  and  Gentner  (2001)  state  that  developing  mental  models  requires  “long¬ 
term  domain  knowledge”  (p.  299).  Given  the  ever  changing  nature  of  DDM  environments,  the 
requirement  for  long-term  domain  knowledge  may  complicate  dynamic  decision  makers’ 
opportunities  to  gain  sufficient  knowledge  to  develop  detailed  and  precise  mental  models  of  DDM 
environments.  However,  that  is  not  to  say  that  dynamic  decision  makes  do  not  develop  mental 
models  of  their  environment  at  all.  It  could  be  that  dynamic  decision  makers  develop  mental 
models  in  a  manner  similar  to  that  of  large-scale  industrial  plant  operators.  Moray  (1996)  states  that 
operators  of  large-scale  industrial  plants  cannot  be  fully  up  to  date  on  the  state  of  the  system  and 
must  rely  on  mental  models  they  have  developed  to  control  the  system.  We  argue  that  there  are 
similarities  between  the  large-scale  industrial  plants  and  DDM  environments.  Both  such  “systems” 
are  complex  in  nature,  can  have  many  variables  for  the  decision  makers  to  control,  and  do  not 
allow  decision  makers  the  ability  to  be  fully  up-to-date  on  the  state  of  the  system.  Given  the 
similarities  between  large-scale  industrial  systems  and  DDM  environments,  it  is  possible  that 
dynamic  decision  makers  develop  mental  models  of  their  environment  in  a  similar  fashion  to  that 
of  operators  controlling  large-scale  systems.  Such  mental  models  would  then  provide  the  dynamic 
decision  maker  with  a  sufficient  understanding  of  the  environment  to  be  able  to  control  the 
environment.  It  is  worth  noting  that  we  did  not  find  any  literature  linking  large  scale  industrial 
system  models  and  DDM. 
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Rate  of  Change.  Over  time,  people  typically  possess  multiple  models  of  complex  systems,  some 
highly  context-bound  and  others  more  abstract  (Gentner,  2002).  However,  there  is  little  indication 
in  the  literature  about  how  quickly  mental  models  are  created.  With  the  rate  of  environmental 
change  for  DDM  tasks  varying  widely  (Brehmer  &  Allard,  1991),  it  is  unclear  whether  or  not  such 
mental  models  would  be  created  fast  enough  to  support  DDM. 

As  can  be  inferred  from  the  above  discussion,  there  is  not  enough  information  at  this  time  to  fully 
understand  the  role  that  mental  models  of  physical  systems  would  play  in  supporting  DDM. 
Information  in  the  literature  that  could  help  understand  such  a  role  would  include  how  feedback  is 
incorporated  into  the  models,  whether  or  not  the  models  are  used  in  relation  to  goals,  how  the 
models  are  created,  and  whether  or  not  they  can  be  applied  in  dynamic  environments. 

5.1.3  Situation  Models 

In  reviewing  the  literature,  the  situation  models  approach  appears  to  be  the  one  that  could  be 
conducive  to  supporting  DDM.  Endsley  (2000a)  describes  situation  models  as  dynamic 
representations  of  a  person’s  knowledge  and  understanding  of  the  present  state  of  a  system. 
Importantly,  situation  models  are  said  to  incorporate  the  dynamics  of  the  system,  the  value  of 
system  parameters,  any  system  changes  over  time,  and  projections  of  future  system  states  (Endsley, 
2000a).  This  means,  in  theory,  that  situation  models  should  be  able  to  support  DDM,  because  these 
models  allow  the  decision  maker  to  incorporate  changes  to  the  system  into  their  situation  model  to 
provide  them  with  an  accurate  and  up-to-date  vision  of  system  status  and  function.  The  decision 
maker  can  then  use  the  situation  model  to  make  future  system  projections  and,  subsequently, 
decisions.  However,  the  available  literature  did  not  provide  details  as  to  whether  or  not  situation 
models  are  developed  in  the  ways  described  by  Endsley. 

In  order  to  evaluate  the  probability  that  situation  models  support  DDM,  we  will  first  assess  how 
well  situation  models  support  the  various  perspectives  of  DDM.  We  will  then  assess  whether  or  not 
situation  models  are  supported  in  DDM  literature. 

Situation  models  lend  some  support  to  the  psychological  perspectives  of  DDM.  In  particular, 
situation  models  are  relevant  for  naturalistic  decision-making  (NDM).  NDM  considers  decision  in 
context-rich  settings  and  the  decision  maker’s  situation  awareness  (Zsambok,  1997;  as  cited  in 
Pliske  &  Klein,  2003).  As  noted  earlier,  Endsley  (2000a)  envisions  situation  models  and  situation 
awareness  as  being  the  same  construct.  Therefore,  a  decision  maker’s  situation  models  would  have 
an  impact  on  the  decisions  made. 

Situation  models  lend  some  support  for  the  control  theory  perspective  of  DDM  as  well.  In  order  to 
control  a  system,  there  are  a  number  of  conditions  that  must  be  met.  These  conditions  include 
having  a  goal,  the  ability  to  control  the  system,  the  ability  to  change  the  state  of  the  system,  and  the 
ability  to  have  a  model  of  the  system  that  describes  what  will  happen  if  changes  are  made  to  the 
system  (i.e.,  to  project  what  the  system  will  look  like  if  changes  are  made;  Brehmer,  1990).  The 
first  3  requirements  are  not  addressed  within  the  situation  model  literature.  However,  the  literature 
suggests  that  situation  models  help  to  fulfil  the  fourth  requirement.  That  is,  the  situation  model  is 
an  up-to-date  model  of  the  system  that  can  be  used  to  make  projections  about  the  future  of  the 
system. 

Although  situation  models  seem  to  be  the  most  consistent  with  DDM  perspectives  discussed  in  this 
report,  the  support  is  somewhat  tenuous.  The  DDM  perspectives  do  not  provide  the  rich  detail 
needed  in  order  to  make  a  stronger  assessment  of  the  usefulness  of  situation  models  in  DDM.  We, 
therefore,  turn  to  the  definitions  of  DDM  to  identify  the  utility  of  situation  models  in  supporting 
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DDM.  To  do  this,  we  first  look  to  the  three  levels  of  situation  models  described  by  Endsley 
(2000b).  These  levels  are: 

1 .  The  observer  needs  basic  perception  of  important  information  in  order  to  form  a  correct 
picture  of  the  situation. 

2.  The  observer  needs  to  be  able  to  comprehend  the  environment/system,  specifically  be  able 
to  integrate  multiple  pieces  of  information  and  determine  their  relevance  to  one’s  goals. 

3.  The  observer  must  be  able  to  project  future  events  from  current  events  and  dynamics. 

In  order  for  situation  models  to  support  DDM,  research  should  show  that  decision  makers  are  able 
to  attain  these  three  levels  of  situation  models. 

The  observer  needs  basic  perception  of  important  information.  As  has  already  been  noted  in  this 
chapter,  we  have  a  number  of  cognitive  biases  restricting  our  ability  to  perceive  important 
information.  For  instance,  we  tend  to  seek  out  information  that  supports  our  current  belief  of  the 
system  (Besnard,  Greathead,  &  Baxter,  2004).  In  addition,  Fu  and  Gonzalez  (2006)  also  suggest 
that  we  do  not  perceive  information  accurately.  As  described  in  section  3.2.1,  Fu  and  Gonzalez’s 
research  using  the  beer  game  found  that  participants  tended  to  ignore  temporal  dynamics  of  a 
system  and  underutilize  information  that  indirectly  influenced  the  outcome  of  their  decisions. 

These  findings  suggest  that  we  are  not  particularly  effective  at  perceiving  important  information 
from  our  environment. 

The  observer  needs  to  be  able  to  comprehend  the  environment/system.  As  with  perception  of 
information,  there  is  reason  to  believe  that  decision  makers  have  difficulty  perceiving  and 
comprehending  their  environment.  This  difficulty  could  be  due  to  the  fact  that  people  have 
difficulty  understanding  temporal  dynamics  and  feedback  delays.  After  conducting  research  on 
time  delays  and  feedback  processes  on  DDM  performance,  Sterman  and  Diehl  (1993)  suggested 
that  poor  performance  on  DDM  tasks  was  due  to  participants’  inability  to  understand  the  system 
because  of  its  complexity.  Follow-up  research  conducted  by  Sterman  (2002;  as  cited  in  Cronin  & 
Gonzalez,  2007)  suggests  we  have  problems  understanding  system  dynamics  because  we  have  a 
poor  understanding  of  stocks,  flows,  and  time  delays.  Not  understanding  these  aspects  of  the 
system,  resulted  in  participants  developing  insufficient  mental  models  to  adequately  support  DDM 
(Cronin  &  Gonzalez,  2007).  Thus,  it  appears  that  dynamic  decision  makers  are  not  particularly 
effective  at  understanding  their  environment  either. 

The  observer  must  be  able  to  project  future  events  from  current  events  and  dynamics.  Research  was 
not  found  that  speaks  to  decision  makers’  ability  to  project  future  events.  However,  it  can  be 
argued  that  if  people  are  unable  to  meet  the  first  two  of  Endsley’s  (2000b)  requirements  in  DDM,  it 
is  doubtful  that  they  would  be  able  to  project  future  events.  That  is,  any  projections  made  about 
future  events  would  be  based  on  limited  information  resulting  in  an  erroneous  mental  model. 

As  outlined  in  section  4.2.3,  Endsley  views  situation  models  as  distinct  from  mental  models. 
Specifically,  mental  models  determine  what  information  is  attended  to,  how  that  information  is 
interpreted  and  integrated,  and  any  projections  made  about  what  will  happen  to  the  system  in  the 
future.  Situation  models,  on  the  other  hand,  are  developed  by  observing  the  world  and  using 
existing  mental  models.  The  information  in  a  mental  model  influences  and  structures  the  situation 
model  data  and  directs  attention,  which  can  be  detrimental  when  the  selection  and  interpretation  of 
information  from  the  mental  model  results  in  a  faulty  SM  (Endsley,  2000b). 

It  is  interesting  to  note  that  although  it  appears  that  situation  models  can  be  of  use  in  DDM, 
research  suggests  that  people  are  not  particularly  effective  at  creating  accurate  situation  models. 
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This  might  be  due  to  the  limited  research  available  to  conduct  this  review.  Future  research  should 
be  undertaken  to  understand  the  ability  of  situation  models  to  support  DDM,  as  well  as  the  ability 
of  dynamic  decision  makers’  ability  to  form  situation  models.  In  particular,  research  should  look  at 
how  observers  perceive  information,  what  are  the  criteria  to  determine  important  information  from 
non-important  information,  how  is  information  perceived  and  sorted  in  dynamic  environments, 
how  well  is  information  integrated  in  dynamic  environments,  and  the  effectiveness  of  system 
predictions  in  dynamic  environments. 

5.1.4  System  Dynamics 

Within  system  dynamics  theory,  much  time  and  effort  has  been  focused  on  mental  models. 
Unfortunately,  inconsistencies  among  the  definition  of  mental  models  in  system  dynamics  makes  it 
difficult  to  clearly  understand  the  role  of  mental  models  in  dynamic  decision  making.  Researchers 
seem  to  agree  that  from  a  system  dynamics  approach,  mental  models  mediate  dynamic  decision 
making  (Cronin  &  Gonzalez,  2007;  Elg,  1996;  Sterman  &  Diehl,  1993).  Further,  an  underlying 
assumption  is  that  improvements  in  one’s  mental  models  automatically  results  in  improved  DDM. 
However,  there  does  not  appear  to  be  supporting  research  clarifying  how  this  actually  takes  place. 
In  fact,  some  research  has  even  demonstrated  a  lack  of  a  direct  relationship  between  mental  models 
and  their  use  in  dynamic  decision  making.  Dhawan  (2006)  showed  that  participants’  mental  models 
improved  after  they  had  training  on  systems  dynamics,  though  there  was  no  change  to  their 
dynamic  decision-making  abilities.  These  results  do  not  support  the  assumption  that  mental 
models  are  completely  intertwined  with  DDM. 

The  means/ends  model  proposed  by  Richardson  et  al.  (1994)  appears  to  provide  the  most  support 
for  the  use  of  mental  models  in  dynamic  decision  making  within  system  dynamics,  and  is  also 
applied  similarly  to  Bryant’s  CECA  model,  though  this  model  does  not  intentionally  take  a  system 
dynamics  approach.  As  is  the  case  with  all  system  dynamics  theory,  this  model  requires  one  to 
have  a  fairly  good  comprehension  about  the  system  and  its  influences  in  order  to  construct  a  useful 
model  that  can  be  used  in  a  “planned  action  setting”.  As  described  earlier,  dynamic  decision 
making  tasks  are  highly  uncertain,  and  are  often  very  ill-structured  problems.  These  DDM  task 
characteristics  are  not  sufficiently  supported  by  system  dynamics  theory,  which  seems  to  require 
more  knowledge  about  the  system  than  is  often  allowed  within  the  characteristics  of  a  DDM  task. 
Uncertainties  within  the  task  may  be  too  great  to  allow  for  a  comprehensive  structure  of  the  system 
to  be  created.  Problems  that  are  poorly  structured,  such  as  those  in  DDM,  may  not  be  comparable 
to  the  systematic  problems  presented  in  much  of  the  system  dynamics  research.  Microworlds  have 
very  systemic,  though  complex,  problems  that  allow  understanding  of  the  system,  predictions, 
actions,  feedback,  repetition  and  learning.  However,  dynamic  decisions  are  often  not  as  well 
defined  in  real  life  situations  as  they  are  in  microworlds. 

System  dynamics  does  however  support  other  DDM  task  characteristics  well.  For  instance,  time  is 
a  key  factor  in  a  DDM  task  and  also  an  important  part  of  a  dynamic  system,  especially  those 
explored  in  research  using  microworld  platforms. 

The  system  dynamics  perspective  also  assumes  that  the  system  will  progress  at  a  rate  that  can  be 
managed  through  improvements  to  a  mental  model  and  that  this  model  can  then  be  used  in  dynamic 
decision  making.  The  DDM  environment  is  very  complex,  can  have  a  high  rate  of  change,  have 
intricate  interdependent  relationships,  and  have  varying  levels  of  quality  of  information.  These 
characteristics  of  DDM  environments  often  do  not  allow  time  to  gain  a  well  organized 
understanding  of  the  system,  its  influences,  or  second  chances  to  correct  or  ameliorate  decision 
actions. 
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The  focus  of  systems  thinking  and  system  dynamics  is  to  study  the  relationships  between 
components  in  the  systems,  especially  feedback  loops  and  the  patterns  of  behaviour  generated  by 
them  (Dhawan,  O’Connor,  &  Borman,  2006).  This  focus  tends  to  be  on  a  single  system,  that 
though  complex,  remains  constant  through  time.  This  means  that  operators  are  asked  to  create  a 
mental  model  of  the  system  and  then  walk  through  the  “states”  of  that  system  along  a  temporal 
dimension.  This  is  distinct  from  DDM,  where  the  system  may  in  fact  be  changing,  requiring  the 
decision  maker  to  not  only  look  at  next  states,  but  also  at  alternative  representations  or  alternative 
mental  models  (Holland  et  ah,  1986). 

Systems  dynamics  does  adequately  support  decision  making  within  stock-and-flow  systems.  The 
extent  to  which  system  dynamics  supports  dynamic  decision  making  outside  of  microworld  and 
system  dynamic  contexts  is  questionable.  Research  looking  at  dynamic  decision  making  within 
systems  dynamics  should  focus  on  how  DDM  task  and  environmental  characteristics  can  be 
supported  by  mental  models,  and  on  the  DDM  contexts  in  which  system  dynamics  research 
transferable. 

5.2  Comparing  requirements  of  DDM  and  Mental  Models 

As  Chapter  3  showed  in  detail,  dynamic  decision-making  involves  (1)  decision  maker(s)  (2)  in  a 
complex  environment  (3)  attempting  to  accomplish  one  or  more  tasks.  A  key  goal  of  this  report  is 
to  explore  how  well  the  use  of  mental  models  would  be  likely  to  support  the  dynamic  decision¬ 
making  process.  The  sections  that  follow  consider  DDM  from  these  three  perspectives  and  explores 
the  extent  to  which  mental  models  might  assist  each  DDM  element.  Each  section  will  be  begin  by 
reminding  the  reader  of  the  DDM  requirements  discussed  in  Section  3,  followed  by  a  discussion 
about  how  well  (or  not)  these  requirements  are  supported  by  mental  models  according  to  the 
literature  and  perspectives  reviewed.  We  will  first  focus  on  the  decision  maker. 

5.2.1  Decision  Maker 

Requirements  of  DDM  -  Dynamic  decision-making  environments  put  a  number  of  demands  on 
decision-makers.  As  noted  in  earlier  sections,  decision  makers  must  sustain  a  workable 
compromise  between  the  demands  of  the  task  and  the  need  to  conserve  one’s  cognitive  resources 
(Brehmer,  2000).  To  do  this,  they  must  handle  the  “core”  decision  task,  and  must  also  control  the 
overall  decision  situation.  Unfortunately,  the  DDM  literature  provides  relatively  little  focus  on 
exactly  how  decision-makers  accomplish  this  task,  but  does  assert  that  this  is  a  critical  requirement 
of  being  a  good  decision  maker. 

Can  Mental  Models  Support  this  Requirement?  -  Given  these  requirements  of  the  dynamic 
decision-maker,  then,  the  key  issue  is  the  extent  to  which  mental  models  are  likely  to  aid  them 
while  they  attempt  to  make  dynamic  decisions. 

The  mental  models  literature  contains  somewhat  more  discussion  about  the  inner  workings  of 
decision  makers,  but  description  of  the  exact  processes  they  use  still  remain  somewhat  vague. 
However,  this  literature  also  suggests  a  number  of  inherent  limitations  of  decision-makers  in  being 
able  to  form  and  apply  mental  models.  These  limitations  suggest  key  areas  in  which  mental  models 
may  not  adequately  support  dynamic  decision  makers. 

Cognitive  limitations.  One  of  the  inherent  problems  in  applying  mental  models  within  the  DDM 
context  relates  to  the  cognitive  limitations  of  the  decision  maker.  As  noted,  DDM  requires  making 
a  series  of  interdependent  decisions  in  a  changing  environment.  Thus,  it  is  critical  that  the  decision 
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maker  is  able  to  perceive  and  interpret  possibly  a  large  amount  of  information  at  a  relatively  fast 
pace.  This  includes  taking  time  into  account,  requiring  the  decision  makers  to  track  temporal 
patterns,  delays  and  the  rate  of  change  of  the  scenario,  all  of  which  are  part  of  the  ‘temporal 
awareness’  of  the  decision  maker.  However,  research  suggests  that  people  can  only  handle  a 
limited  amount  of  information  in  terms  of  volume  and  in  terms  of  number  of  variables  (e.g., 
Forrester,  1994;  as  cited  in  Doyle  &  Ford,  1998).  This  is  particularly  problematic  for  DDM,  as 
DDM  tasks  take  place  in  complex  environments  in  which  the  decision  maker  is  required  to  observe 
and  interpret  many  changing  characteristics.  The  inability  to  relate  only  a  limited  number  of 
variables  suggests  that  a  decision  maker  may  not  be  able  to  generate  a  mental  model  that  is 
adequately  developed  to  include  all  the  details  of  a  DDM  environment. 

Further  complicating  the  decision  maker’s  ability  to  form  adequate  mental  models  is  the  size  of  the 
model  that  would  need  to  be  created.  In  order  to  conduct  dynamic  decision  tasks  that  are  constantly 
changing,  for  example,  decision  makers  would  need  to  update  their  mental  model  every  time  they 
received  new  information  in  order  to  keep  the  model  current.  This  would  only  be  possible  if  new 
information  came  in  at  a  pace  that  allowed  the  decision  maker  to  incorporate  the  new  information 
effectively  into  the  current  mental  model.  This  could  result  in  the  decision  maker  developing  a 
larger  than  manageable  mental  model  and/or  having  to  devote  an  undue  amount  of  attention  to 
maintaining  an  accurate  model. 

Another  limitation  noted  in  the  mental  model  literature  is  that  it  is  unclear  whether  decision  makers 
are  aware  about  whether  their  mental  models  are  complete  or  not.  Unfortunately,  this  issue  is  not 
discussed  prominently.  However,  given  a  range  of  research  and  theory  related  to  decision-making, 
there  is  some  evidence  that  decision  makers  are  often  more  confident  in  their  decisions  than  they 
should  be  (Tversky  &  Kahneman,  1974).  They  may  also  show  the  same  overconfidence  bias  in 
their  mental  models.  Again,  this  is  likely  to  be  problematic  in  the  dynamic  decision-making 
environment,  in  which  the  costs  of  inaccurate  estimations  could  be  serious. 

Mental  effort.  Although  building  and  maintaining  mental  models  are  core  activities  in  the  control 
of  dynamic  complex  systems,  dynamic  decision  makers  are  required  to  conduct  a  host  of 
operations  (e.g.,  plan  actions,  control  movements),  all  of  which  use  cognitive  resources.  This 
suggests  that  mental  models  are  more  likely  to  be  helpful  when  decision  makers  are  motivated  to 
create  and  use  these  models. 

Subsequently,  decision  makers  are  often  required  and/or  motivated  to  save  resources  where  they 
can.  This  may  mean  that  mental  models  are  built  based  on  partial  pieces  of  evidence  (Besnard, 
Greathead,  &  Baxer,  2004).  As  a  consequence,  decision  makers  may  settle  on  a  solution  that  they 
deem  to  be  good  enough  even  though  it  may  be  suboptimal.  This  is  problematic  as  the 
consequences  of  flawed  mental  models  can  be  disastrous  when  operating  within  complex  dynamic 
system. 

There  are  other  ways  noted  in  the  literature  in  which  dynamic  decision  makers  try  to  reduce  their 
mental  efforts  as  well.  For  example,  operators  in  critical  systems  sometimes  would  rather  lose 
some  situation  awareness  rather  than  spend  time  gathering  data,  which  could  mean  a  loss  of  control 
of  the  dynamic  situation  (Besnard,  Greathead,  &  Baxter,  2004).  Therefore,  operators  save  resources 
when  possible  by  using  mechanisms  such  as  selective  memory  and  heuristics.  This  also  means  that 
they  build  mental  models  that  contain  partial  rather  than  complete  information.  That  is,  they 
satisfice  rather  than  optimize  and  settle  on  a  solution  that  they  deem  to  be  “good  enough’’. 

(Besnard,  Greathead,  &  Baxter,  2004).  Markham  and  Gentner  (2001)  also  argue  that  people  shift  to 
using  learned  rules  rather  than  mental  models  for  reasoning  as  they  gain  experience  with  a  domain. 
With  such  experience,  people  are  able  to  access  their  stored  knowledge  of  the  outcome  rather  than 
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carry  out  full  mental  simulations  of  the  behaviour.  This  may  mean  that  when  they  have  experience 
within  a  given  domain,  decision  makers  may  tend  to  use  heuristics  rather  than  mental  models  to 
make  dynamic  decisions. 

Heuristics  and  biases.  Optimal  dynamic  decisions  require  a  consistently  critical  stance  to  the 
emergence  of  new  approaches  and  creative  solutions.  For  example,  seeking  disconfirming  evidence 
can  indicate  ways  in  which  one’s  conceptual  model  is  not  an  accurate  representation  of  the 
situation.  In  many  situations,  this  kind  of  information  is  more  valuable  than  confirmatory  evidence 
(Bryant,  2004).  However,  one  of  the  problems  with  mental  models  is  that  they  are  not  necessarily 
conducive  to  giving  adequate  weight  to  new  information.  Sustaining  accurate  mental  models  can  be 
cyclical.  Specifically,  the  current  mental  model  can  assist  the  collection  of  information,  and  that 
information  is  then  used  to  alter  the  mental  model.  However,  as  argued  by  Besnard,  Greathead,  and 
Baxter  (2004),  we  seek  out  information  that  supports  our  current  belief  of  the  situation.  If  the 
dynamic  decision  maker  already  has  a  flawed  mental  model  of  the  situation,  the  new  information 
selected  and  integrated  by  the  decision  maker  will  be  the  information  that  supports  current  beliefs 
(often  in  contrast  to  more  accurately  calibrated  beliefs).  Subsequently,  this  new  information  will 
just  make  the  mental  model  more  erroneous.  With  the  pace  of  many  dynamic  decision-making 
environments  being  relatively  fast,  it  seems  likely  that  the  “pull”  to  stick  with  safe  and  established 
models  will  often  be  stronger  than  taking  the  time  to  revise  those  models  to  include  new  (and 
potentially  more  accurate)  information. 

5.2.2  DDM  Environments 

Requirements  of  DDM  -  The  environments  within  which  DDM  occurs  have  a  number  of  distinct 
features.  They  are  characterized  by  high  levels  of  complexity,  and  the  sheer  number  of  diverse 
goals  often  in  play.  There  are  also  often  many  different  forms  of  feedback  that  are  relevant,  and 
delays  within  the  feedback  systems  can  impose  additional  demands  on  decision  makers. 

Can  Mental  Models  Support  this  Requirement?  -  The  key  issue,  however,  is  whether  (and  if  so, 
how)  the  mental  model  construct  might  help  manage  these  issues. 

Complexity.  Brehmer  and  Allard  (1991)  define  the  complexity  of  the  situation  as  relative  to  the 
capacity  for  the  decision  maker  to  control  the  number  of  processes,  goals,  action  alternative,  and 
side  effects  of  the  system.  This  suggests  that  the  bigger  the  scope  of  the  decision-making 
environment,  the  more  information  the  decision  maker  would  be  required  to  perceive  and  integrate. 
For  example,  a  military  forces  commander  may  be  required  to  identify  the  mission  goals,  identify 
the  resources  available  to  achieve  the  goals,  anticipate  the  enemy’s  actions,  anticipate 
consequences  of  given  orders  given,  etc.  As  already  noted,  decision  makers  have  a  limited 
cognitive  capacity  with  respect  to  perceiving  and  integrating  information.  Therefore,  the  more 
complex  the  DDM  environment,  the  less  likely  that  the  decision  maker  will  be  able  to  develop  a 
detailed  and  accurate  model. 

Besnard,  Greathead  and  Baxter  (2004)  argue  that  mental  models  of  DDM  tasks  are  likely  only 
correct  at  early  stages  of  the  interaction.  They  state  that  interactions  evolve  with  time  and  the 
situation  may  degrade  (e.g.,  due  to  an  emergency).  The  mental  model  then  gets  simplified  and 
becomes  based  on  correlations  between  system  elements  rather  than  new  information. 

Accepting  the  notion  of  pervasive  limitations  in  human  cognitive  capacity,  the  broader  or  bigger 
the  scope  that  the  decision  maker  must  consider  logically  implies  less  depth  or  detail  that  can  be 
included.  This  ‘breadth’  versus  ‘depth’  relationship  is  not  new  to  studies  of  cognition,  but  the 
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effects,  tradeoffs  and  optimizations  and  how  individuals  can  find  the  best  balance  do  not  appear  to 
have  been  explored  in  detail  in  either  the  mental  model  or  DDM  literature. 

An  additional  nuance  of  complexity  likely  to  impact  on  the  ability  to  use  mental  models  for  DDM 
relates  to  the  natural  levels  of  coherence  and  order  within  a  given  situation.  There  is  a  clear  sense  in 
the  mental  model  literature  that  one  of  the  important  functions  of  these  models  is  that  they  promote 
coherence,  the  ability  to  understand  even  diverse  linkages  among  elements.  Mental  models  do  not 
necessarily  have  to  be  wholly  consistent,  one  might  argue,  but  they  have  to  make  sense  of  how 
diverse  elements  might  fit  together.  In  a  sense,  mental  models  can  help  to  represent  a  complex 
“story”  in  a  way  that  makes  sense.  The  potential  problem  within  DDM  environments,  on  the  other 
hand,  is  that  these  environments  are  not  necessarily  conducive  to  creating  coherence  from  disorder, 
and  for  understanding  the  causal  structure  among  diverse  elements.  This  suggests  that  mental 
models  may  be  more  difficult  to  form  in  DDM  environments. 

Feedback.  Denzau  and  Roy  (2005)  state  that  the  best  conditions  under  which  to  form  mental 
models  are  those  in  which  there  is  frequent  feedback  about  the  situation.  This  suggests  that  when 
dynamic  decision-makers  encounter  situations  that  provide  timely  and  accurate  feedback,  they  will 
be  able  to  develop  the  appropriate  mental  model.  This  could  assist  their  subsequent  interactions 
with  the  “system”  and  make  them  more  effective,  because  mental  models  help  to  simplify  the 
environment.  However,  decision  makers  are  not  always  able  to  learn  the  true  models  in  situations 
in  which  feedback  is  infrequent  or  non-existent  (Denzau  &  Roy,  2005).  However,  the  DDM 
literature  lists  feedback  delays  as  a  characteristic  of  DDM  environments.  Such  delays  in  feedback 
are  problematic  for  forming  mental  models  (indeed  for  learning  in  general)  because  people  have 
trouble  dealing  with  long  time  delays  between  actions  and  feedback  (Fu  &  Gonzalez,  2006; 

Sterman  &  Diehl,  1993).  Subsequently,  decision  makers  will  have  difficulties  incorporating  the 
delayed  information  into  their  mental  models.  The  impact  of  this  is  that  decision  makers  will  not  be 
able  to  form  true  mental  models  of  a  dynamic  situation  (Denzau  &  Roy,  2005). 

According  to  Bryant  (2004),  it  is  not  the  amount  of  data  that  is  the  paramount  when  forming  mental 
models,  but  rather  the  quality  and  value  of  the  information  to  the  issue  at  hand.  However,  the 
quality  of  the  feedback  information  in  dynamic  decision  environments  is  described  as  variable 
(Brehmer  &  Allard,  1991).  Therefore,  when  dynamic  decision  makers  receive  high  quality 
feedback,  such  information  will  support  their  mental  models.  On  the  other  hand,  when  dynamic 
decision  makers  receive  poor  quality  feedback,  such  information  will  lessen  the  validity  of  their 
mental  models. 

The  ordering  of  the  feedback  also  has  an  impact  on  the  ability  to  create  mental  models.  Zwaan 
(1999)  states  that  people  prefer  to  incorporate  information  in  chronological  order.  Therefore, 
information  that  is  presented  outside  of  chronological  order  is  hard  to  comprehend  and  integrate 
into  an  accurate  mental  model.  In  complex  DDM  environments,  feedback  is  provided  to  decision 
makers  when  and  where  available.  With  no  guarantee  that  feedback  will  be  provided  with 
chronological  accuracy,  decision  makers  might  have  difficulty  incorporating  new  information.  This 
feature  of  DDM  environments  may  promote  inaccurate  or  incomplete  mental  models. 

5.2.3  DDM  Tasks 

Requirements  of  DDM  -  The  DDM  literature  outlines  several  key  characteristics  of  DDM  tasks. 

As  noted  earlier,  such  tasks  are  often  characterized  by  being  highly  interdependent,  involving  high 
levels  of  uncertainty  and  having  time  as  an  important  element.  Such  tasks  are  often  ill-structured,  in 
the  sense  that  it  may  be  difficult  for  decision  makers  to  fully  identify  all  the  relevant  features  of  a 
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given  problem.  These  features  of  DDM  tasks  are  likely  to  impact  on  the  usefulness  of  mental 
models  from  several  perspectives. 

Can  Mental  Models  Support  this  Requirement?  -  As  a  whole,  the  nature  of  typical  DDM  tasks  is 
likely  to  pose  challenges  for  the  use  of  mental  models. 

One  reason  for  such  challenges  is  due  to  time.  As  noted  earlier,  time  is  a  key  factor  in  many  DDM 
tasks.  That  is,  DDM  decisions  need  to  be  made  in  real  time,  which  means  that  decision  makers  do 
not  always  have  control  over  when  and  where  to  make  the  decisions  (Brehmer  &  Allard,  1991).  A 
potential  problem  with  creating  mental  models,  however,  is  that  they  take  time  and  energy  to  form, 
and  it  cannot  be  assumed  that  a  person  entering  a  new  situation  will  be  able  to  spontaneously  form 
a  new  and  unfamiliar  mental  model.  The  literature  seems  to  argue  that  this  might  be  possible  if  they 
enter  a  somewhat  familiar  situation  or  one  with  which  they  have  had  previous  experience. 

However,  if  time  is  a  factor  (as  it  is  in  DDM  environments)  forming  a  usable  mental  model  may  be 
precluded. 

Where  mental  models  are  likely  to  help  in  very  time  limited  situations  is  where  they  already  exist 
and  can  be  brought  to  bear  on  the  situation  at  hand.  This  is  evident  in  the  literature  comparing  the 
performance  of  experts  and  novices.  Experts  are  often  assumed  to  be  faster  because  they  have 
functional  mental  models  that  help  them  to  perform  tasks  in  a  more  timely  fashion.  However, 
according  to  Endsley  (2000b),  time  is  also  a  key  factor  in  the  formation  of  a  situation  model.  When 
using  a  situation  model,  it  is  critical  to  know  how  much  time  is  left  until  some  event  occurs  or  until 
an  action  must  be  taken.  Knowing  the  rate  at  which  the  situation  is  changing  allows  for  the 
projection  of  future  situations.  Similarly,  the  literature  also  suggests  that  additional  aspects  of  time 
must  also  be  included  in  the  mental  model.  This  includes  knowing  the  present  time,  how  long  it 
takes  for  actions  to  show  their  effects  (feedback),  recognizing  changes  in  the  world,  and 
recognizing  when  an  updated  mental  model  is  required. 

The  dynamic  decision  environment  is  also  inherently  uncertain:  the  environment  changes 
autonomously  and  as  a  result  of  the  decision  maker’s  actions  and  the  system  has  aspects  that  are 
not  visible  to  the  decision  maker  (Brehmer,  1992).  This  feature  of  DDM  tasks  is  likely  to  have 
negative  implications  for  the  decision  maker’s  ability  to  form  mental  models.  As  stated  by  Denzau 
and  Roy  (2005),  the  best  conditions  in  which  to  form  mental  models  are  relatively  simple  situations 
in  which  all  or  most  of  the  information  is  known.  The  description  of  the  ideal  context  for  the 
formation  of  mental  models  most  prominent  in  the  available  literature  is  that  of  a  clearly  bounded 
problem  space,  in  which  the  entire  problem  is  transparent,  known,  and  amenable  to  seeing  clear 
linkages  among  the  component  parts  of  the  problem  space.  DDM  tasks,  on  the  other  hand,  are  often 
highly  ambiguous,  containing  multiple  interdependent  parts  and  having  high  levels  of  uncertainty 
around  each  part.  This,  of  course,  does  not  mean  that  mental  models  cannot  be  helpful,  simply  that 
the  limitations  of  whatever  models  can  be  formed  should  be  recognized. 

5.3  Conclusion 

The  goal  of  this  review  was  to  attempt  to  understand  the  nature  of  mental  models  in  dynamic 
decision-making  and  the  role  that  they  play  in  this  complex  process.  Although  there  are  certainly 
many  different  ways  in  which  mental  models  are  likely  relevant  to  DDM,  the  literature  in  this  area 
is  unfortunately  at  a  relatively  early  stage  of  development,  and  is  not  currently  mature  enough  to 
provide  a  frill  answer.  This  is  true  from  both  the  perspective  of  mental  models  in  general  and  in 
relation  to  dynamic  decision-making. 
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In  terms  of  mental  models,  although  there  is  good  agreement  about  the  potential  importance  of 
mental  models,  there  is  lihle  consistency  in  the  instantiation  of  this  concept,  so  much  so  that  the 
existence  of  mental  models  is  assumed  as  the  starting  point  with  little  true  description  of  what  the 
views  of  mental  models  actually  are.  Ironically,  one  of  the  dangers  of  the  mental  model  concept  is 
that  it  seems  to  have  had  such  intuitive  appeal  that  researchers  and  theorists  seem  to  assume  that 
everyone  else  shares  the  same  definition  of  the  construct  that  they  have.  More  so,  effectively 
capturing  mental  models  still  remains  elusive.  There  is  agreement  that  attempts  to  measure  mental 
models  may  in  the  process  change  them,  and  that  what  information  is  captured  may  or  may  not  be 
related  to  a  decision  maker’s  mental  model  at  all. 

Within  the  realm  of  dynamic  decision-making,  it  seems  reasonable  to  look  for  ways  in  which 
decision  makers  might  be  able  to  simplify  their  environment,  and  to  posit  that  perhaps  mental 
models  might  be  one  way  in  which  they  might  do  this.  If,  as  some  have  argued,  people  are  able  to 
form  models  of  the  workings  of  complex  physical  systems,  they  might  also  be  able  to  use  similar 
processes  to  understand  and  to  make  sense  of  dynamic  environments.  Certainly,  there  are  some  key 
linkages  that  show  the  potential  role  of  mental  models  in  DDM.  On  the  whole,  however,  there  are 
also  many  ways  in  which  the  current  approaches  to  understanding  mental  models  may  be  too 
underspecified  to  be  able  to  manage  the  complexities  of  decision-making  in  the  DDM 
environments.  Perhaps  not  surprisingly,  we  would  argue  that  the  more  constrained  situation  model 
approach  is  the  most  viable  current  candidate  for  being  able  to  serve  the  requirements  of  the  DDM 
environment.  Hopefully,  further  development  of  this  area  will  assist  the  development  of  clear 
linkages  among  the  mental  model  and  DDM  domains. 

An  underlying  assumption  throughout  the  dynamic  decision-making  approaches  is  that  there  is  a 
link  between  mental  models  and  the  quality  of  decision  making;  however,  research  suggesting 
improvements  to  mental  models  results  in  improved  decision  making  was  not  found.  While  a  lack 
of  support  does  not  mean  that  the  assumption  is  incorrect,  it  does  question  the  assertion.  Further 
exploration  on  the  link  between  improvements  in  both  aspects  should  be  empirically  explored 
(though  it  is  acknowledged  that  determining  the  nature  of  the  relationship  is  confounded  by  the 
aforementioned  issues  with  measuring  mental  models). 

A  number  of  inherent  limitations  in  decision  makers  forming  mental  models  (i.e.  limited  working 
memory,  biases,  erroneous  mental  models,  etc)  suggest  key  areas  in  which  mental  models  may  not 
adequately  support  dynamic  decision-makers.  Compounding  these  limitations  are  complex  DDM 
environments  with  delayed  (at  best)  feedback.  Since  the  literature  is  consistent  in  finding  that 
mental  models  are  often  erroneous  under  the  most  ideal  circumstances,  it  is  reasonable  to  assert 
that  the  DDM  conditions  would  only  add  to  those  errors. 

Nonetheless,  further  exploration  of  the  nature  of  both  mental  models  and  their  potential  intersection 
with  DDM  will  hopefully  be  possible  as  research  and  theory  progress,  and  as  the  literature  matures. 
The  section  that  follows  explores  a  range  of  possible  future  questions  and  issues  that  will  further 
this  goal. 
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6.  Future  Research  Recommendations 
based  on  Gaps  in  the  Literature 


This  section  includes  two  tables  that  summarize  gaps  in  the  literature  that  have  been  touched  on  in 
the  previous  sections  of  the  report.  The  first  table  will  look  at  what  mental  models  research  would 
have  to  explore  to  better  serve  DDM  and  the  second  table  looks  at  mental  models  research 
questions  in  general. 


Table  4:  Mental  Models  and  DDM 


Research  Issue 

Research  Questions 

Limits  to  mental  models 

How  are  mental  models  formed  when  the  system  variables  are 
interdependent?  How  do  mental  models  incorporate  interdependent 
variables? 

Can  mental  models  incorporate  information  when  the  system  is  changing 
quickly?  Is  there  a  limit  to  the  rate  of  change?  Do  decision  makers  use  more 
than  one  mental  model  in  such  cases? 

Can  mental  models  be  formed  when  the  decision  maker  is  lacking  clear  goals? 

Can  mental  models  be  developed  in  complex  environments?  How  useful 
would  such  models  be? 

To  what  extent  do  mental  models  improve  dynamic  decision  making?  In 
dynamic  decision  making  tasks,  what  constitutes  effective  mental  models? 

Once  familiar  with  a  system,  do  experts  replace  mental  models  with  other 
cognitive  structures? 

How  do  inconsistent  models  impact  on  DDM  effectiveness?  How  do  dynamic 
decision  makers  choose  between  inconsistent  mental  models  when  making 
dynamic  decisions?  What  are  the  criteria  for  choosing  between  inconsistent 
models?  When  do  people  realize  they  have  inconsistent  mental  models?  How 
is  it  that  people  register  that  they  have  inconsistent  mental  models? 

How  is  information  perceived  and  sorted  into  mental  models  in  dynamic 
environments? 

Can  people  have  multiple  mental  models  of  the  same  system,  at  different 
abstraction  levels? 

How  different  are  models  of  the  environment  and  models  of  physical  systems, 
especially  since  the  latter  should  form  part  of  the  former? 

Awareness  of  mental  models 

Are  dynamic  decision  makers  aware  of  their  mental  models  of  the  system? 

Can  they  be  made  more  aware? 

What  skills  do  dynamic  decision  makers  need  to  ensure  they  form  accurate 
mental  models?  Are  there  ways  for  decision  makers  to  become  aware  of  their 
mental  models  inaccuracies? 

To  what  extent  is  the  decision  maker  ‘a  part  of  the  mental  model?  How  do 
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decision  makers  to  know  when  the  ‘best  time’  is  to  make  a  decision?  Or  are 
decisions  made  only  when  they  must  be? 

Are  dynamic  decision  makers  aware  of  their  perception  biases  and  how  they 
impact  on  their  mental  models?  Are  there  questions  decision  makers  can  ask 
themselves  to  reduce  the  effects  of  these  errors? 

To  what  extent  do  decision  makers  consciously  run  their  mental  models? 

Feedback 

Can  dynamic  decision  makers  be  trained  to  better  incorporate  feedback  delays 
into  their  mental  models? 

Do  mental  models  incorporate  feed  forward  loops?  If  so,  how  useful  are  they  in 
making  dynamic  decisions? 

How  does  feed  forward  information  impact  mental  models? 

Can  mental  models  be  altered  to  better  incorporate  feedback  or  feed  forward 
information? 

Individual  differences 

Are  there  individual  differences  that  make  some  dynamic  decision  makers 
better  at  formulating  mental  models?  Can  these  attributes  be  taught  to  other 
dynamic  decision  makers? 

Table  5.  Mental  Models 


Research  Issue 

Research  Questions 

Mental  models  in  general 

What  are  the  links  between  the  different  types  of  mental  models?  Are  there 
actually  different  types,  or  are  they  all  part  of  the  same  cognitive  construct? 

Are  there  aspects  of  creating  and  updating  mental  models  that  are  readily 
learned? 

How  quickly  are  mental  models  developed? 

How  are  mental  models  different  from  any  knowledge  stored  in  the  brain? 

Are  mental  models  transitory  or  more  permanent? 

Is  there  a  point  where  mental  models  become  too  unwieldy? 

What  Is  the  role  of  deductive  versus  inductive  reasoning  in  mental  models 
errors? 

Measuring  mental  models 

What  are  the  most  effective  methods  for  measuring  mental  models? 

Is  the  effectiveness  of  the  measurement  method  dependent  on  the  situation? 

Can  measurement  methods  be  combined  together  effectively,  resulting  in  a 
more  coherent  understanding? 
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(U)  The  complex  and  dynamic  nature  of  various  types  of  operations  pose  specific  cognitive 
challenges  on  the  decision-making  process  that  the  current  training  regiment  of  military 
commanders  does  not  directly  address.  Therefore,  DRDC  Toronto  is  interested  in 
researching  training  techniques  to  prepare  Canadian  Forces  (CF)  commanders  and  staff 
for  decision-making  in  such  complex  and  dynamic  environments  (12sk).  This  report 
provides  a  review  of  relevant  DDM  literature  and  mental  models  literature  as  it  relates  to 
DDM. 

DDM  consists  of  (1)  decision  maker(s)  (2)  in  a  complex  environment  (3)  attempting  to 
accomplish  one  or  more  tasks.  DDM  is  required  in  environments  with  high  risk  and 
complexity,  and  involves  the  performance  of  tasks  requiring  multiple  steps,  that  are 
inherently  time  sensitive,  interdependent,  and  which  exert  influence  over  the  surrounding 
environment  as  well  as  being  influenced  by  it.  Dynamic  decision-making  has  been 
explored  from  different  perspectives,  including  systems  theory,  psychology,  and  control 
theory  from  the  engineering  domain.  These  perspectives  put  varying  amounts  of  focus  on 
different  aspects  of  DDM.  What  is  common  to  all  of  these  approaches  are  the  assumptions 
that  whether  forming  models  of  complex  systems  or  making  intuitive  decisions  based  on 
very  little  information,  people  tend  to  form  some  sort  of  mental  model  to  undertake  DDM. 

At  a  broad  level,  mental  models  can  be  described  as  personal  mental  representations  of 
our  world.  Although  there  is  no  one  agreed  definition  of  mental  models,  they  are  generally 
recognized  to  serve  three  key  functions:  to  describe,  to  predict,  and  to  explain  our  world. 
The  aim  of  this  report  was  to  explore  how  mental  models  are  understood  across  the 
propositional  logic,  physical  systems,  situation  model,  and  system  dynamics  perspectives. 
Few  similarities  in  the  descriptions  of  mental  models  were  found  between  the  domains 
reviewed  in  this  report.  In  addition,  mental  models  were  found  to  be  subject  to  a  range  of 
shortcomings  that  impact  the  accuracy  of  mental  models  (e.g.,  limited  cognitive  capacity, 
erroneous  causal  links,  based  on  incomplete  information,  systemic  errors). 

The  extent  to  which  DDM  could  be  supported  by  each  of  the  unique  approaches  to 
understanding  mental  models  was  assessed,  as  well  as  an  exploration  of  how  well  the  use 
of  mental  models  would  be  likely  to  support  the  dynamic  decision-making  process.  This 
review  closes  with  an  outline  of  the  mental  models  research  that  needs  to  be  better 
explored  to  serve  DDM,  as  well  as  an  outline  of  the  general  mental  models  research 
questions  that  should  be  addressed  in  future  research. 

(U)  Le  caractere  complexe  et  dynamique  de  certains  types  d’operations  pose  des  problemes 
cognitifs  particuliers  au  processus  de  prise  de  decisions  que  I’instruction  actuelle  des 
commandants  militaires  ne  traite  pas  directement.  RDDC  Toronto  est  done  a  la  recherche 
de  techniques  d’instruction  qui  permettront  de  former  les  commandants  et  les  membres 
d’etat  major  des  Forces  canadiennes  (FC)  a  la  prise  de  decisions  dans  des 
environnements  complexes  et  dynamiques  (12sk).  Le  present  compte  rendu  renferme  un 
apergu  de  la  documentation  sur  la  PDD  et  sur  les  modeles  mentaux  associes  a  la  PDD. 

La  PDD  implique  la  participation  (1)  d’un  decideur,  (2)  dans  un  environnement  complexe, 
(3)  qui  tente  d’executer  une  ou  plusieurs  taches.  La  PDD  est  necessaire  dans  des 
environnements  complexes  ou  le  risque  est  eleve.  Elle  implique  I’execution  de  taches 
requerant  des  etapes  multiples,  pour  lesquelles  le  temps  est  important,  qui  sont 
interdependantes  et  qui  exercent  une  influence  sur  le  milieu  environnant,  tout  en  etant 
influencees  par  lui.  La  prise  de  decisions  dynamique  a  ete  exploree  a  partir  de  differents 
points  de  vue,  notamment  la  theorie  des  systemes,  la  psychologie  et  la  t  heorie  de 


controle  dans  le  domaine  de  I’ingenierie.  Ces  points  de  vue  mettent  I’accent,  a  divers 
degres,  sur  differents  aspects  de  la  PDD.  Line  hypothese  est  commune  a  toutes  ces 
approches  :  qu’il  s’agisse  de  former  des  modeles  de  systemes  complexes  ou  de  prendre 
des  decisions  intuitives  a  partir  de  tres  peu  de  renseignements,  les  gens  ont  tendance  a 
developper  une  certaine  forme  de  modele  mental  pour  effectuer  une  PDD. 

De  fagon  generale,  on  peut  decrire  les  modeles  mentaux  comme  des  representations 
mentales  de  notre  monde.  Bien  qu’il  n’y  ait  aucun  consensus  sur  la  definition  des  modeles 
mentaux,  ces  derniers  sent  generalement  reconnus  pour  etre  associes  a  trois  functions 
cles  :  decrire,  predire  et  expliquer  le  monde  qui  nous  entoure.  Le  present  compte  rendu  a 
pour  but  d’explorer  comment  les  modeles  mentaux  sent  compris  du  point  de  vue  de  la 
logique  des  propositions,  des  systemes  causals,  du  modele  de  situation  et  de  la 
dynamique  des  systemes.  On  a  trouve  peu  de  similitudes  dans  les  descriptions  des 
modeles  mentaux  entre  les  domaines  examines  dans  la  presente  analyse.  En  outre,  on  a 
decouvert  que  des  modeles  mentaux  etaient  sujets  a  une  variete  de  lacunes  influengant 
leur  exactitude  (p.  ex.,  capacite  cognitive  limitee,  liens  causals  errones  fondes  sur  des 
renseignements  incomplets,  erreurs  systemiques). 

On  a  evalue  jusqu’a  quel  point  la  PDD  pouvait  etre  appuyee  par  chacune  des  approches 
uniques  a  la  comprehension  des  modeles  mentaux,  de  meme  qu’on  a  explore  jusqu’a  quel 
point  I’utilisation  de  modeles  mentaux  appuie  le  processus  de  prise  de  decisions 
dynamique.  L’examen  se  termine  sur  les  aspects  de  la  recherche  sur  les  modeles 
mentaux  devant  etre  mieux  explores  pour  servir  la  PDD,  de  meme  que  sur  un  apergu  des 
questions  generales  relatives  a  la  recherche  sur  les  modeles  mentaux  qui  devraient  etre 
traitees. 
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